SteveProXNA

PyGame Programming Setup

noreply@blogger.com (SteveProXNA) — Tue, 15 Sep 2026 08:00:15 +0000

In 2020, we checked out Python Setup Cheat Sheet to create development environment. However, working through code exercises to improve code skills is tedious; we would like to build video games using PyGame!

Let’s check it out!

PyGame
PyGame is a free and open-source cross-platform library for the development of video games using Python. Using Simple DirectMedia Layer to abstract most common functions makes it easier to write these programs.

Tutorials
In order to build video games in Python we first need to learn PyGame APIs. Here are some great resources:

PyGame Docs	Comprehensive documentation reference for using Pygame to create games
PyGame Examples	GitHub repository with sample programs that demonstrate how to use Pygame
PyGame Coders	Beginner-friendly Pygame tutorial that teaches the game development basics
PyGame Tutorial	Extremely thorough Pygame tutorial used to drive examples in this blog post

Example I - Hello PyGame
All examples listed are Python 3.11.11 compatible. Launch PyCharm | New Project. Enter the following info:

Location:	~/HelloPyGame
Interpreter type:	uv
Python version:	3.11
Path to uv:	~/.local/bin/uv

PyCharm should setup UV virtual environment and configure Python interpreter if not then enter commands:

  uv venv --python 3.11
  source .venv/bin/activate           # OR .\.venv\Scripts\activate
  which python
  `which python` --version            # Python 3.11.11

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  uv lock

  uv sync

Create the following file: main.py. Enter the command uv run main.py. Verify Hello PyGame is working OK!

Example I - Hello PyGame

  import pygame
  pygame.init()
  
  screen = pygame.display.set_mode((640, 480))
  pygame.display.set_caption("Hello PyGame!")
  
  font = pygame.font.Font(None, 48)  # Built-in default font
  text = font.render("Hello PyGame!", True, (255, 255, 255))
  
  running = True
  while running:
      for event in pygame.event.get():
          if event.type == pygame.QUIT:
              running = False
  
      screen.fill((0, 0, 0))
      screen.blit(text, (220, 220))
      pygame.display.flip()
  
  pygame.quit()

Example II - Hello Splash
Repeat previous exercise but render splash screen. Launch PyCharm | New Project. Enter the following info:

Location:	~/HelloSplash
Interpreter type:	uv
Python version:	3.11
Path to uv:	~/.local/bin/uv

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  uv venv --python 3.11
  source .venv/bin/activate           # OR .\.venv\Scripts\activate
  which python
  `which python` --version            # Python 3.11.11
  uv sync

Create the following file: main.py. Enter the command uv run main.py. Verify splash screen renders OK!

Example II - Hello Splash

  import pygame
  pygame.init()
  
  screen = pygame.display.set_mode((640, 480))
  pygame.display.set_caption("Hello Splash")
  sms_surface = pygame.Surface((640, 480))
  
  splash = pygame.image.load("StevePro.bmp").convert()
  
  running = True
  while running:
      for event in pygame.event.get():
          if event.type == pygame.QUIT:
              running = False
  
      wide: int = (640 - splash.get_width()) / 2
      high: int = (480 - splash.get_height()) / 2
      sms_surface.blit(splash, dest=(wide, high))
  
      screen.fill((0, 0, 0))
      scaled = pygame.transform.scale(sms_surface, (640, 480))
      screen.blit(scaled, (0, 0))
  
      pygame.display.flip()
  
  pygame.quit()

Example III - Hello Text
Repeat previous exercise but render font text. Launch PyCharm | New Project. Setup project as previously:

Example III - Hello Text

  import pygame
  pygame.init()
  
  screen = pygame.display.set_mode((640, 480))
  pygame.display.set_caption("Hello Text")
  sms_surface = pygame.Surface((640, 480))
  
  font = pygame.font.Font("emulogic.ttf", 20)
  running = True
  while running:
      for event in pygame.event.get():
          if event.type == pygame.QUIT:
              running = False
  
      sms_surface.fill((0, 0, 0))
      text_surface = font.render("Hello Text!", False, (255, 255, 255))
      sms_surface.blit(text_surface, (0, 0))
      scaled = pygame.transform.scale(sms_surface, (640, 480))
      screen.blit(scaled, (220, 220))
  
      pygame.display.flip()
  
  pygame.quit()

Example IV - Hello Sprites
Repeat previous exercise but render out sprites. Launch PyCharm | New Project. Setup project previously:

Example IV - Hello Sprites

  import pygame
  pygame.init()
  
  screen = pygame.display.set_mode((640, 480))
  pygame.display.set_caption("Hello Sprites")
  sms_surface = pygame.Surface((320, 240))
  
  spritesheet = pygame.image.load("Sprites.bmp").convert()
  
  def get_sprite(sheet, col, row, width=32, height=32):
      rect = pygame.Rect(col * width, row * height, width, height)
      return sheet.subsurface(rect)
  
  running = True
  while running:
      for event in pygame.event.get():
          if event.type == pygame.QUIT:
              running = False
  
      sms_surface.fill((0, 0, 0))
  
      sprite1 = get_sprite(spritesheet, col=0, row=0)
      sprite2 = get_sprite(spritesheet, col=0, row=1)
      sms_surface.blit(sprite1, (64, 64))
      sms_surface.blit(sprite2, (128, 64))
  
      scaled = pygame.transform.scale(sms_surface, (640, 480))
      screen.blit(scaled, (0, 0))
  
      pygame.display.flip()
  
  pygame.quit()

Example V - Hello Input
Repeat previous exercise but with input detection. Launch PyCharm | New Project. Setup project previously:

Example V - Hello Input

  import pygame
  pygame.init()
  
  screen = pygame.display.set_mode((640, 480))
  pygame.display.set_caption("Hello Input")
  
  font = pygame.font.Font(None, 48)  # Built-in default font
  text = font.render("Hello Input - Press space...", True, (255, 255, 255))
  
  running = True
  while running:
      for event in pygame.event.get():
          if event.type == pygame.QUIT:
              running = False
  
      screen.fill((0, 0, 0))
      screen.blit(text, (100, 220))
  
      keys = pygame.key.get_pressed()
      if keys[pygame.K_SPACE]:
          space = font.render("Space pressed!!", True, (255, 255, 255))
          screen.blit(space, (200, 260))
  
      pygame.display.flip()
  
  pygame.quit()

Example VI - Hello Sound
Example VII - Hello Music
Example VIII - Hello Audio
Repeat previous exercise but integrate all audio. Launch PyCharm | New Project. Setup project previously:

Example VIII - Hello Audio

  import pygame
  pygame.init()
  pygame.mixer.init()
  
  screen = pygame.display.set_mode((640, 480))
  pygame.display.set_caption("Hello Audio")
  
  font = pygame.font.Font(None, 48)  # Built-in default font
  sound_01 = pygame.mixer.Sound("right.wav")
  sound_02 = pygame.mixer.Sound("wrong.wav")
  pygame.mixer.music.load("music.mp3")
  
  def render_text(text, color=(255, 255, 255)):
      return font.render(text, False, color)
  
  text1 = render_text("Hello Audio!" )
  text2 = render_text("Press Left - sound #1")
  text3 = render_text("Press Right - sound #2")
  text4 = render_text("Press Up - play music")
  text5 = render_text("Press Down - stop music")
  
  running = True
  while running:
      for event in pygame.event.get():
          if event.type == pygame.QUIT:
              running = False
  
      screen.fill((0, 0, 0))
      screen.blit(text1, (220, 140))
      screen.blit(text2, (160, 180))
      screen.blit(text3, (160, 220))
      screen.blit(text4, (160, 260))
      screen.blit(text5, (160, 300))
  
      keys = pygame.key.get_pressed()
      if keys[pygame.K_LEFT]:
          sound_01.play()
      if keys[pygame.K_RIGHT]:
          sound_02.play()
      if keys[pygame.K_UP]:
          pygame.mixer.music.play()
      if keys[pygame.K_DOWN]:
          pygame.mixer.music.stop()
  
      pygame.display.flip()
  
  pygame.quit()

Example III - Hello Text	Example IV - Hello Sprites

Example V - Hello Input	Example VIII - Hello Audio

Summary
To summarize, we now have the building blocks to implement PyGame APIs; we are in an excellent position to reuse existing video game content and port video game from another language to Python using PyGame! This will be the topic of the next post.

PyTorch Setup Cheat Sheet II

noreply@blogger.com (SteveProXNA) — Tue, 05 May 2026 08:02:00 +0000

In the previous post, we checked out PyTorch Setup as an open-source machine learning framework widely used for building + training neural networks. Here we installed and configured our nVidia driver and CUDA Toolkit to build some simple PyTorch programs. However, we built PyTorch3d + TorchSparse packages from source each time. Now we'd like to pre-build custom wheels housed on localhost for future examples reuse.
Let's check it out!

Example I
All examples listed are Python 3.10.19 compatible. Launch PyCharm | New Project. Enter the following info:

Location:	~/HelloPyTorch3dWheels
Interpreter type:	uv
Python version:	3.10
Path to uv:	~/.local/bin/uv

PyCharm should setup UV virtual environment and configure Python interpreter if not then enter commands:

  uv venv --python 3.10
  source .venv/bin/activate           # OR .\.venv\Scripts\activate
  which python
  `which python` --version            # Python 3.10.19

In the PyCharm Terminal | Enter the following commands for UV to install packages to build custom wheels:

  uv pip install twine
  uv pip install numpy==1.26.4
  uv pip install --index-url https://download.pytorch.org/whl/cu121 \
      "torch==2.2.0+cu121" "torchvision==0.17.0"  "torchaudio==2.2.0"

Create directory to house custom wheels mkdir -p wheelhouse-cu121. Write shell scripts to code wheel logic:

PyTorch3D	chmod +x steveprobuild_pytorch3d_wheel.sh
TorchSparse	chmod +x steveprobuild_torchsparse_wheel.sh

Execute shell scripts to build custom wheels. These will be the custom wheels used locally on next examples:

COMMAND	ARTIFACT
bash steveprobuild_pytorch3d_wheel.sh	stevepropytorch3d-0.7.7-cp310-cp310-linux_x86_64.whl
bash steveprobuild_torchsparse_wheel.sh	steveprotorchsparse-2.0.0b0-cp310-cp310-linux_x86_64.whl

Example II
Repeat previous exercise using locally built wheels. Launch PyCharm | New Project. Enter the following info:

Location:	~/HelloPyTorch3dWheels
Interpreter type:	uv
Python version:	3.10
Path to uv:	~/.local/bin/uv

Copy custom wheels built from previous example: mkdir wheelhouse-cu121. Copy files into new directory:

  mkdir -p wheelhouse-cu121
  cp -r ../01-Example/wheelhouse-cu121 .

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  export MAX_JOBS=1
  export NVCC_THREADS=1

  uv lock
  uv sync

Create the following file: main.py. Enter the command uv run main.py. Verify PyTorch packages installed!

Example II

  Hello PyTorch3d
  torch 2.2.0+cu121
  pytorch3d 0.7.7
  cuda 12.1
  cuda True
  torch_geometric 2.7.0
  torch_scatter 2.1.2+pt22cu121
  torch_sparse 0.6.18+pt22cu121
  torch_cluster 1.6.3+pt22cu121
  torch_spline_conv 1.2.2+pt22cu121
  torchsparse 2.0.0b

Example III
Repeat previous exercise but upload custom wheels. Launch PyCharm | New Project. Enter the following info

Location:	~/HelloPyTorch3dWheels
Interpreter type:	uv
Python version:	3.10
Path to uv:	~/.local/bin/uv

Copy custom wheels built from previous example: mkdir wheelhouse-cu121. Copy files into new directory:

  mkdir -p wheelhouse-cu121
  cp -r ../01-Example/wheelhouse-cu121 .

In the PyCharm Terminal | Enter the following commands to install devpi packages to upload wheels locally:

  uv pip install devpi-server
  uv pip install devpi-client

  uv pip install devpi-web
  uv pip install twine

Launch Terminal #1. Initialize devpi-server on localhost port 3141 and start. Navigate to the homepage URL

  devpi-init					# Launch browser
  devpi-server --host 127.0.0.1 --port 3141	# Navigate http://localhost:3141

Launch Terminal #2. Connect to the devpi-client and login. Create custom cuda-wheels index and activate

  devpi use http://localhost:3141
  devpi login root --password=''

  devpi index -c cuda-wheels bases=root/pypi  
  devpi use root/cuda-wheels

Finally upload 2x custom wheel files to devpi-web available at http://localhost:3141/root/cuda-wheels

  twine upload --repository-url http://localhost:3141/root/cuda-wheels/ wheelhouse-cu121/*

Example IV
Repeat previous exercise using uploaded wheels. Launch PyCharm | New Project. Enter the following info:

Location:	~/HelloPyTorch3dWheels
Interpreter type:	uv
Python version:	3.10
Path to uv:	~/.local/bin/uv

In the PyCharm Terminal | Enter the following commands to install devpi packages to consume local wheels:

  uv pip install devpi-server
  uv pip install devpi-client

  uv pip install devpi-web
  uv pip install twine

Launch Terminal #1. Initialize devpi-server on localhost port 3141 and start. Navigate to the homepage URL

  devpi-init					# Launch browser
  devpi-server --host 127.0.0.1 --port 3141	# Navigate http://localhost:3141

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  export MAX_JOBS=1
  export NVCC_THREADS=1

  uv lock
  uv sync

Create the following file: main.py. Enter the command uv run main.py. Verify PyTorch packages installed!

Example IV

  Hello PyTorch3d
  torch 2.2.0+cu121
  pytorch3d 0.7.7
  cuda 12.1
  cuda True
  torch_geometric 2.7.0
  torch_scatter 2.1.2+pt22cu121
  torch_sparse 0.6.18+pt22cu121
  torch_cluster 1.6.3+pt22cu121
  torch_spline_conv 1.2.2+pt22cu121
  torchsparse 2.0.0b

Example V
Repeat previous exercise but wrap logic as Azure ML endpoint. Launch PyCharm | New Project. Enter info:

Location:	~/HelloAzureML
Interpreter type:	uv
Python version:	3.10
Path to uv:	~/.local/bin/uv

PyCharm should setup UV virtual environment and configure Python interpreter if not then enter commands:

  uv venv --python 3.10
  source .venv/bin/activate           # OR .\.venv\Scripts\activate
  which python
  `which python` --version            # Python 3.10.19

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  export MAX_JOBS=1
  export NVCC_THREADS=1

  uv lock
  uv sync

Create the following directories app and tests. Enter all code for app/scoring.py. Test all code using pytest.
Launch terminals. Enter following commands: build Docker image run Docker container and submit request:

Terminal #1

  docker build -t azml-gpu-local:latest .
  docker run --gpus all -p 5001:5001 azml-gpu-local:latest

Terminal #2

  curl --location --request POST 'http://localhost:5001/score' \
    --header 'Content-Type: application/json' \
    --data-raw '{
    "points": [
      [0.0, 0.0, 1.0],
      [0.1, 0.0, 0.99],
      [-0.1, 0.0, 0.99],
      [0.0, 0.1, 0.99],
      [0.0, -0.1, 0.99],
      [0.7, 0.7, 0.0],
      [-0.7, 0.7, 0.0],
      [0.7, -0.7, 0.0],
      [-0.7, -0.7, 0.0],
      [0.0, 0.0, -1.0]
    ]
  }'

Example V

  {
      "num_vertices": 1524,
      "num_faces": 2892,
      "mean_pixel": 0.9926620125770569,
      "image_shape": [
  		128,
  		128,
  		3
  	],
      "device": "cuda:0"
  }

Cleanup as necessary: stop and remove the docker process esp. if that process is continued to be in use:

  docker stop $(docker ps -q)
  docker rm -f $(docker ps -q)

Summary
To summarize, we have built PyTorch3d and TorchSparse packages from source, pre-built the custom wheels housed on localhost for examples reuse. We are now in an excellent position: drive Deep Learning education!

PyTorch Setup Cheat Sheet

noreply@blogger.com (SteveProXNA) — Sat, 04 Apr 2026 09:00:00 +0000

PyTorch is an open-source machine learning framework widely used for building + training neural networks. While PyTorch provides the core building blocks for tensors, neural networks, and training loops, most real-world machine learning problems involve domain-specific data—images, audio, graphs + also 3D geometry!
Let's check it out!

PyTorch ecosystem extends specialized libraries for different domains [vision, audio, 3D, sparse data] go-to choice for everything from prototyping research ideas to deploying production-grade deep learning systems.

PACKAGES

torch	Core PyTorch library providing tensors, autograd and neural network building blocks
torchaudio	Tools + datasets for audio processing, waveform transformations and speech models
torchvision	Computer Vision utilities for datasets: pretrained models and image transformations
pytorch3d	3D Deep Learning library for working with meshes, point clouds rendering pipelines
torchsparse	High-performance sparse tensor library for 3D geometry e.g. LiDAR and voxel grids
Torch-scatter	Optimized operations for scattering and aggregating neural networks tensor values
Torch-sparse	Sparse matrix operations for PyTorch + often used in 3D Geometric Deep Learning
Torch-cluster	Clustering and neighborhood learning query operations e.g. kNN and radius graphs
Torch-spline-conv	Spline-based convolution layers for Geometric Deep Learning on graphs and meshes

CUDA
CUDA [Compute Unified Device Architecture] is a proprietary parallel computing platform and API that allows software to use GPUs for accelerated general-purpose processing in high-performance scientific computing.

Therefore, to leverage PyTorch computations on GPUs one must install nVidia GPU drivers that support your CUDA version, install CUDA Toolkit that includes nvcc compiler with all PyTorch compatible built packages.

Development setup example: Ubuntu 20.04 | Linux kernel 5.15.0-9 | nVidia GPU Driver v535 | CUDA 12.1

INSTALLATION
Complete nVidia CUDA Toolkit installation guide found here but can be summarized in the following steps:

  uname -r				# 5.15.0-139-generic

Step #1 - Remove partial CUDA installs

  sudo apt remove -y cuda* nvidia-cuda-toolkit
  sudo apt autoremove -y
  sudo rm -rf /usr/local/cuda*

Step #2 - Install nVidia Driver 535

  sudo apt update
  sudo apt install -y nvidia-driver-535
  sudo reboot

Step #3 - Verify nVidia Driver and CUDA version - nvidia-smi

Step #4 - Install CUDA 12.1 Toolkit

  sudo mkdir -p /etc/apt/keyrings
  
  curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/3bf863cc.pub \
   | sudo gpg --dearmor -o /etc/apt/keyrings/cuda-archive-keyring.gpg
  
  echo "deb [signed-by=/etc/apt/keyrings/cuda-archive-keyring.gpg] \
  https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /" \
  | sudo tee /etc/apt/sources.list.d/cuda.list
  
  sudo apt update

Step #5 - Configure Environment Variables - source ~/.bashrc

  export CUB_HOME=~/cub
  export CUDA_HOME=/usr/local/cuda-12.1
  export PATH=$CUDA_HOME/bin:$PATH
  export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

Step #6 - Verify CUDA 12.1 Toolkit - nvcc --version

Kernel	5.15.0-139
Driver	nvidia-driver-535
CUDA 12.1	/usr/local/cuda-12.1
CUDA toolkit	/usr/local/cuda-12.1/bin/nvcc

Example I
All examples listed are Python 3.10.19 compatible. Launch PyCharm | New Project. Enter the following info:

Location:	~/HelloPyTorch3d
Interpreter type:	uv
Python version:	3.10
Path to uv:	~/.local/bin/uv

PyCharm should setup UV virtual environment and configure Python interpreter if not then enter commands:

  uv venv --python 3.10
  source .venv/bin/activate           # OR .\.venv\Scripts\activate
  which python
  `which python` --version            # Python 3.10.19

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  export MAX_JOBS=1
  export NVCC_THREADS=1

  uv lock
  uv sync

Create the following file: main.py. Enter the command uv run main.py. Verify PyTorch packages installed!

Example I

  Hello PyTorch3d
  torch 2.2.0+cu121
  pytorch3d 0.7.7
  cuda 12.1
  cuda True
  torch_geometric 2.7.0
  torch_scatter 2.1.2+pt22cu121
  torch_sparse 0.6.18+pt22cu121
  torch_cluster 1.6.3+pt22cu121
  torch_spline_conv 1.2.2+pt22cu121
  torchsparse 2.0.0b

Example II
Repeat previous exercise but wrap logic as Azure ML endpoint. Launch PyCharm | New Project. Enter info:

Location:	~/HelloAzureML
Interpreter type:	uv
Python version:	3.10
Path to uv:	~/.local/bin/uv

PyCharm should setup UV virtual environment and configure Python interpreter if not then enter commands:

  uv venv --python 3.10
  source .venv/bin/activate           # OR .\.venv\Scripts\activate
  which python
  `which python` --version            # Python 3.10.19

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  export MAX_JOBS=1
  export NVCC_THREADS=1

  uv lock
  uv sync

Create the following files: app/scoring.py and .env. Enter the command which azmlinfsrv. Edit configurations Copy and paste the azmlinfsrv path as the script value. Enter the following info to launch Azure ML endpoint:

script value	~/HelloAzureML/.venv/bin/azmlinfsrv
Parameters	--entry app/scoring.py
Working directory	~/HelloAzureML
Path to .env files	~/HelloAzureML/.env

Alternatively, launch terminals. Enter following commands to launch Azure ML endpoint and submit request:

Terminal #1	set -a; source .env; set +a ~/.venv/bin/azmlinfsrv --entry app/scoring.py
Terminal #2	curl --location --request POST 'http://localhost:5001/score' \ --header 'Content-Type: application/json' \ --data-raw '{}'

Example II

  {
      "message": "Hello PyTorch3d",
      "torch_version": "2.2.0+cu121",
      "pytorch3d_version": "0.7.7",
      "cuda_version": "12.1",
      "cuda_available": true,
      "torch_geometric_version": "2.7.0",
      "torch_scatter_version": "2.1.2+pt22cu121",
      "torch_sparse_version": "0.6.18+pt22cu121",
      "torch_cluster_version": "1.6.3+pt22cu121",
      "torch_spline_conv_version": "1.2.2+pt22cu121",
      "torchsparse_version": "2.0.0b"
  }

Cleanup as necessary: find and kill processes using TCP port 5001 esp. if that port is continued to be in use:

  sudo fuser -k 5001/tcp		# Linux
  sudo lsof -ti tcp:5001		# MacOS

Summary
To summarize, we have installed and configured our nVidia driver and CUDA Toolkit and built some simple PyTorch programs. However, we have built the PyTorch3d + TorchSparse packages from source each time. Ideally, we would like to pre-build custom wheels once and house on localhost for future examples reuse!
This will be the topic of the next post.

PyBind Setup Cheat Sheet

noreply@blogger.com (SteveProXNA) — Tue, 17 Mar 2026 09:00:00 +0000

In 2024, we checked out OpenAI Retro Cheat Sheet as an open-source project that provides an interface to interact with various retro video games for purpose of Reinforcement Learning research. This project: using C/C++ for high-performance but leverages pybind11 to create bindings for Python client code consumption.

Let's check it out!

pybind11
A lightweight header-only library that can be used to integrate C++ with Python to create bindings exposing C++ functions to Python. Client code written in Python can be consumed to invoke the underlying C++ code.

Installation
All examples here are executed on Ubuntu Linux. Therefore install pybind11 globally to begin the examples:

 sudo apt-get update
 sudo apt-get install pybind11-dev
 sudo apt install build-essential g++

Example I
Create an example that exposes C++ function to Python with pybind11. Launch terminal | Enter commands:

  mkdir -p ~/HelloPyBind
  cd HelloPyBind
  python  -m venv .venv
  source .venv/bin/activate           # OR .\.venv\Scripts\activate
  which python
  `which python` --version            # Python 3.8.10
  pip install pybind11
  pip install --upgrade pip

Create the following files: example.cpp, setup.py, test.py. Enter the following C++ and Python source code:

example.cpp

  #include <pybind11/pybind11.h>
  
  int add(int x, int y)
  {
      return x + y;
  }
  
  PYBIND11_MODULE(example, m)
  {
      // optional module docstring
      m.doc() = "pybind11 example plugin";
      m.def("add", &add, "A function which adds two numbers");
  }

setup.py

  from setuptools import setup, Extension
  import pybind11
  
  ext_modules = [
      Extension(
          "example",
          ["example.cpp"],
          include_dirs=[pybind11.get_include()],
          language="c++"
      ),
  ]
  
  setup(
      name="example",
      version="0.1",
      ext_modules=ext_modules,
  )

test.py

  import example
  
  result = example.add(1, 2)
  print(f"1 + 2 = {result}")

Build C++ code using setup.py build inplace. Finally execute python test.py for Python to execute C++ code!

  python setup.py build_ext --inplace		# example.cpython-38-x86_64-linux-gnu.so
  python test.py				# OUTPUT	1 + 2 = 3

Example II
Repeat previous exercise but prefer PyCharm IDE. Launch PyCharm | New Project. Enter the following info:

Location:	~/HelloPyBind
Interpreter type:	uv
Python version:	3.11
Path to uv:	~/.local/bin/uv

PyCharm should setup UV virtual environment and configure Python interpreter if not then enter commands:

  uv venv --python 3.11
  source .venv/bin/activate           # OR .\.venv\Scripts\activate
  which python
  `which python` --version            # Python 3.11.11

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  uv add pybind11
  uv add setuptools

  uv lock
  uv sync

Create the following files: example.cpp, setup.py, test.py. Enter C++ and Python code similar to Example I. Build C++ code using setup.py build install. Finally execute uv run test.py for Python to execute C++ code!

  uv run setup.py build		
  uv run setup.py install		# example.cpython-38-x86_64-linux-gnu.so
  uv run test.py			# OUTPUT	3 + 5 = 8	9 - 5 = 4

Example III
Repeat previous exercise but prefer CMake to build C++ code via CMakeLists.txt. Create PyCharm Project:

Location:	~/HelloPyBind
Interpreter type:	uv
Python version:	3.11
Path to uv:	~/.local/bin/uv

In the PyCharm Terminal | Enter the following commands for UV to install and sync package dependencies:

  uv add pybind11

  uv sync

Create the following files: example.cpp, CMakeLists.txt, test.py. Enter code similar to Example II but update:

CMakeLists.txt

  cmake_minimum_required(VERSION 3.16)
  project(example)
  
  # Find the Python 3.11-specific pybind11 CONFIG from pip
  execute_process(
          COMMAND ${Python3_EXECUTABLE} -m pybind11 --cmakedir
          OUTPUT_VARIABLE pybind11_DIR
          OUTPUT_STRIP_TRAILING_WHITESPACE
  )
  find_package(pybind11 REQUIRED CONFIG)
  pybind11_add_module(example example.cpp)

Build C++ code using cmake and make. In the PyCharm Terminal | Enter the following commands to build:

  mkdir -p build
  cd build

  cmake -DPython3_EXECUTABLE=$(which python) ..
  make -j$(grep -c ^processor /proc/cpuinfo)

Enter commands to copy library to be used. Finally execute uv run test.py for Python to execute C++ code!

  python -c "import sysconfig, shutil, glob;		\		
  dst = sysconfig.get_paths()['platlib'];		\
  so = glob.glob('*.so')[0];				\
  shutil.copy2(so, dst)"
  cd ..
  uv run test.py			# OUTPUT	Hello, World!

IMPORTANT
The first 3x examples worked but tightly coupled Python and C++ without being able to debug separately!

Example IV
Repeat previous exercise but prefer to modify the project layout to separate top level Python and C++ code:

  ~/HelloPyBind/
  ├── cpp/
  │   ├── src/
  │   │   ├── api/
  │   │   │   ├── my_api.h
  │   │   │   └── my_api.cpp
  │   │   ├── bindings/
  │   │   │   └── pybind_module.cpp       # pybind11 bindings
  │   │   ├── CMakeLists.txt
  │   │   └── main.cpp                    # C++ executable entry point
  │   ├── tests/
  │   │   ├── CMakeLists.txt
  │   │   └── test_api.cpp
  │   └── CMakeLists.txt                  # top-level C++ (CLion entry point)
  │
  └── python/
      ├── .venv/
      │   └── lib/
      │       └── python3.11/
      │           └── site-packages/
      │               └── my_api_py.cpython-311-x86_64-linux-gnu.so
      ├── test.py
      ├── pyproject.toml
      └── README.md

Create PyCharm Project. Setup virtual environment as before then create CLion project to build C++ code.

Location:	~/HelloPyBind/python
Interpreter type:	uv
Python version:	3.11
Path to uv:	~/.local/bin/uv

Launch CLion | New Project. Create C++ Executable using C++ 17. Enter the following CLion information:

C++	C++ Executable
Location:	~/HelloPyBind/cpp
Language standard:	C++17

Set build directory in CLion. File menu | Settings... | Build, Execution, Deployment | CMake | Build directory

Setup folder layout as above. Enter all C++ source code and tests. Rebuild entire solution in Debug mode.

IMPORTANT
CMakeLists.txt files are configured to copy shared object SO file into the Python .venv virtual environment

Launch PyCharm | Complete the test runner. Finally execute uv run test.py for Python to execute C++ code!

  uv run test.py				# OUTPUT	1 + 2 = 3

Example V
Repeat previous exercise but prefer more complexity to build C++ code with classes as consumed by Python

Create PyCharm Project. Setup virtual environment as before then create CLion project to build C++ code. Launch CLion | New Project. Create C++ Executable using C++ 17. Enter the following CLion from before.

Set build directory in CLion. File menu | Settings... | Build, Execution, Deployment | CMake | Build directory. Setup folder layout as above. Enter all C++ source code and tests. Rebuild entire solution in Debug mode.

Launch PyCharm | Complete the test runner. Finally execute uv run test.py for Python to execute C++ code!

  uv run test.py		# OUTPUT	
  # Guitar: 'Fender' [6-string] = $1500.0
  # Guitar: 'Ibanez' [7-string] = $1200.0
  # Guitar: 'Gibson' [6-string] = $2400.0

Example VI
Repeat previous exercise but prefer more complexity to build C++ code with templates consumed by Python

Create PyCharm Project. Setup virtual environment as before then create CLion project to build C++ code. Launch CLion | New Project. Create C++ Executable using C++ 17. Enter the following CLion from before.

Set build directory in CLion. File menu | Settings... | Build, Execution, Deployment | CMake | Build directory. Setup folder layout as above. Enter all C++ source code and tests. Rebuild entire solution in Debug mode.

Launch PyCharm | Complete the test runner. Finally execute uv run test.py for Python to execute C++ code!

  # OUTPUT	
  # Container[0] = 0.0
  # Container[1] = 1.0
  # Container[2] = 2.0
  # Container[3] = 3.0
  # Container[4] = 4.0

  # OUTPUT	
  # Container[5] = 5.0
  # Container[6] = 6.0
  # Container[7] = 7.0
  # Container[8] = 8.0
  # Container[9] = 9.0

Example VII
Repeat previous exercise but prefer Visual Studio 2022 on Windows to build an increasing C++ code base:

  ~/HelloPyBind/
  ├── cpp/
  │   ├── src/
  │   │   ├── core/			  # Core API implementation
  │   │   │   ├── *.h
  │   │   │   └── *.cpp
  │   │   ├── math/			  # Math-related API
  │   │   │   ├── *.h
  │   │   │   └── *.cpp
  │   │   ├── mesh/			  # Mesh-related API
  │   │   │   ├── *.h
  │   │   │   └── *.cpp
  │   │   ├── bindings/
  │   │   │   └── pybind_module.cpp       # pybind11 bindings
  │   │   ├── CMakeLists.txt
  │   │   └── main.cpp                    # C++ executable entry point
  │   ├── tests/
  │   │   ├── CMakeLists.txt
  │   │   ├── test_matrix.cpp
  │   │   ├── test_vector.cpp
  │   │   ├── test_mesh.cpp
  │   │   ├── test_mesh_algorithms.cpp
  │   │   └── test_mesh_processor.cpp
  │   └── CMakeLists.txt                  # top-level C++ (CLion entry point)
  └── python/

Launch Visual Studio 2022 | Continue without code. File | Open | CMake... Navigate to cpp/CMakeLists.txt. Build menu | Build All. Finally, choose Test menu | Test Explorer. Run All Tests in View or choose to Debug:

Summary
To summarize, we have demonstrated various PyBind examples in which C++ library code is consumed by a single Python API exclusively. However, in future there may be instances in which the C++ library may need to be consumed by multiple languages. In this case ctypes.cdll.LoadLibrary() may be better that PyBind!

Python Package Cheat Sheet

noreply@blogger.com (SteveProXNA) — Tue, 03 Feb 2026 10:00:00 +0000

In 2020, we checked out Python Setup Cheat Sheet as an interpreted high-level programming language runs on Windows, Mac OS/X and Linux using pip as the de facto standard package-management system. However while this worked, requirements.txt is brittle with package dependencies + versioning. Poetry was introduced to resolve deterministic builds but at slower dependency resolution. Enter uv for blazing speed + reliability J

Let's check it out!

History
Python Setup Cheat Sheet detailed how to install pip as the de facto standard package-management system on Windows, Mac OS/X and Linux. However, requirements.txt does not lock down transitive dependencies + versioning which becomes brittle. Poetry solved this problem using pyproject.toml configuration and lock file.

uv
Replicating Poetry with more deterministic builds using using pyproject.toml configuration and lock file, uv is built in Rust by Astral as a full rethinking of Python packaging designed for speed and simplicity. uv: pitched as "A single tool to replace pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv and more". Here is a detailed article comparing the Python Packaging Landscape: Pip vs. Poetry vs. UV from a developer's standpoint.

IMPORTANT
First check out the traditional way using python and pip to compare and illustrate benefits of now using uv:

Virtual Environment
A virtual environment isolates Python project interpreter and installed packages from the system and other Python projects which means each project should have its own environment, its version and dependencies.

Create a virtual environment in the traditional way using python and pip then activate virtual environment:

python -m venv .venv
Linux OR Mac OS/X	source .venv/bin/activate
Windows	.\.venv\Scripts\activate

Next, install packages using python, pip and requirements.txt OR poetry with pyproject.toml configuration:
Brittle and/or slow [transitive] dependency resolution!

Installation
Download and install uv for Linux, Mac OS/X or Windows OR Launch PyCharm and install from home page:

Linux	curl -LsSf https://astral.sh/uv/install.sh \| sh
Mac OS/X	brew update && brew install uv
Windows	powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 \| iex"

uv init
Create new directory and execute following commands to initialize Python project with an interpreter version

Launch Terminal | Execute the following commands:

  mkdir HelloUV
  cd HelloUV
  uv init --python 3.11.11

uv venv
Navigate into Python project and execute following commands to create the virtual environment and activate

Launch Terminal | Execute the following commands:

  uv venv --python 3.11.11
  source .venv/bin/activate		# Windows: .\.venv\Scripts\activate

uv python
At this point you have Python project initialized with virtual environment activated. Confirm correct version of Python interpreter installed and activate. When using PyCharm ensure the IDE interpreter path is aligned!

Inside PyCharm Terminal | Execute the following commands:

  which python
  `which python` --version

uv add
Install Python packages either using uv pip install or uv add commands. Prefer uv add because this updates pyproject.toml file automagically thus you are able to execute uv sync command to install the dependencies.

Inside PyCharm Terminal | Execute the following commands:

  uv pip list
  uv add requests
  uv sync

uv tree
After execute uv add you can verify what is installed by checking pyproject.toml file or execute uv pip list but another useful method is uv tree which shows hierarchy of all your project's dependencies and relationships.

Inside PyCharm Terminal | Execute the following command: uv tree

uv sync
Execute uv add or update pyproject.toml to install dependencies. Execute uv sync to update environment.

uv lock
The uv.lock file records all the exact versions of all project dependencies UV figures are compatible from the pyproject.toml file to ensure constant deterministic builds with the same dependenices each time and CI/CD

uv tool
UV tool installs persistent tools into virtual environment that are required for that particular Python project:

Inside PyCharm Terminal | Execute the following commands:

  uv tool list
  uv tool install ruff
  uv tool run ruff check
  uv tool upgrade --all
  uv tool uninstall ruff
  uv tool list

uvx tool
Finally uvx is an alias for uv tool run but is designed to be run immediately without installing it persistently:

Inside PyCharm Terminal | Execute the following command: uvx ruff check

uv cache
When you use uv all tools and dependencies are stored in cache. These commands remove all cached data:

  uv cache clean
  rm -r "$(uv python dir)"
  rm -r "$(uv tool dir)"

Commands
Here is a quick summary of popular uv commands used during workflow. A comprehensive list can be found.

  uv init --app				# Scaffold project
  uv python install 3.11		# Install Python
  uv venv --python 3.11			# Create virtual environment
  uv add requests			# Add dependencies
  uv add -D pytest			# Add dev dependencies
  uv sync --frozen			# Sync (locked)
  uv sync				# Sync (normal)
  uv run python main.py			# Run program
  uv run python -V			# Show Python
  uvx ruff check .			# Run tools ad‑hoc
  uv lock				# Update lockfile

Docker
A well-built uv Docker image simplifies deployment + ensures application runs consistently in environment:

  FROM python:3.11.11-slim
  # Install uv
  COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
  COPY . /app
  WORKDIR /app
  # Install dependencies and clear cache
  RUN uv sync --no-dev --frozen
  RUN rm -rf ~/.cache/uv
  CMD ["python", "-c", "print('Hello, World!')"]

This is an example Dockerfile snippet how to integrate uv! But build and run execution with the commands:

  docker build -t uv-hello .
  docker run --rm uv-hello

GitHub Actions
A common GitHub Actions pattern is to use UV to install only Prod dependencies during your build or deploy:

  name: Testing
  on:
    push:
      branches:
        - "main"
  jobs:
    build:
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v4
        - name: Install UV
          run: curl -LsSf https://astral.sh/uv/install.sh | sh
        - name: Sync only prod dependencies
          run: uv sync --no-group dev --group prod
        - name: Run tests
          run: uv run pytest

Summary
To summarize, uv has been pitched "One Tool for Everything" as uv replaces pip, virtualenv, pip-tools, pipx, poetry, pyenv, twine and is fast at every step. However, this has only scratched the surface as uv integrates well with other tools for fast reproducible Python workflows such as direnv, pre-commit hooks, pdm + more!

uv	Handles Python installs, environment creation, and dependency resolution with locking
direnv	Evaluates .envrc managing environment variables automatically applied to shell session
pdm	Orchestrate workflows + provide unified tooling to build, version, and publish pacakges

Retrospective XVII

noreply@blogger.com (SteveProXNA) — Thu, 01 Jan 2026 09:00:00 +0000

Last year, I conducted a simple retrospective for 2024. Therefore, here is a retrospective for year 2025.

2025 Achievements

Document DevOps managed clusters provisioning setup E.g.: Azure AKS, AWS-EKS, GCP-GKE
Present + report The Evolution of Software Deployment Cloud CI/CD setup theory to practical
Consolidate pytest unit test framework features such as fixtures, factories, mocking, patching
Transition Python package management from pip to poetry and finally to uv for blazing speed
Introduction to MLOps theory learning to practical ML models and deployment implementation
Extend GitLab CI/CD experience to upskill to GitHub Actions executing on localhost using ACT
Resurrect PyBind knowledge on OpenAI Retro to Mac on CLion for Python to C/C++ examples
Reverse Engineer RetroAchievements project open source code for Simpsons Trivia integration

Note: Reverse Engineering and setup RetroAchievements project open source code is a big achievement!

2026 Objectives

Document Python package management move from pip to poetry to uv for new Python projects
Continue MLOps integration to bridge Software Engineering to Machine Learning knowledge gap
Register PyBind Python to C/C++ setup cross platform and explore debugging across languages
Harness prior OpenAI Retro and RetroAchievements experience and apply to future AI projects!

AI Demand
According to 2025 analysis global AI and ML Engineer job openings numbering in the hundreds of thousands with employers increasingly willing to hire based on demonstrable skills rather than advanced degrees. One market study forecasts AI Engineering sector could grow from $17.4 billion in 2025 to $87.5 billion by 2030 implying a sustained growth rate for the next 5yrs.

Therefore, as a Professional Software Engineer how are some ways to transition to a more focused AI-role?

MLOps Engineer
By expanding traditional Software Development Life Cycle skills across DevOps as modern delivery practices have evolved one next step is the progression into MLOps: enabling the deployment monitoring and the mgt of machine learning models built by dedicated ML Engineers. This provides immediate value to the AI-driven teams while supporting ongoing machine learning expertise.

AI Engineer
An AI Engineer specializes in building integrating deploying and optimizing machine + deep learning systems bridging the gap between research models and production-ready software. A Software Engineer focuses on: application logic architecture performance and code quality whereas an AI Engineer works at the intersection of Software Engineering and Data Science.

Future
In 2024 we checked out OpenAI Gym and Retro to provide early exposure to AI-centric principles integrating Reinforcement Learning environments using Python and C/C++ to upskill AI concepts such as agent training environment design reproducibility and iterative experimentation.

In 2025 we checked out RetroAchievements ecosystem to further deepen these skills dealing with emulation internals memory inspection and state tracking all highly relevant to RL and model-driven control systems.

Together this experience forms connection between long-standard Professional Software Engineer expertise and the emerging demands of AI Engineering. Combining systems programming with complex environment instrumentation and practical RL-work aligns with one of the fastest-growing roles in the technology sector!

Simpsons Trivia Retro Achievements

noreply@blogger.com (SteveProXNA) — Sat, 15 Nov 2025 09:00:00 +0000

In 2018, Simpsons Trivia built for the Sega Master System was one entry in the annual SMS Power! Coding competition. The immediate forum feedback super positive. In 2020, member FeRcHuLeS posted enjoyment of the game and mastered on retroachievements.org. I was unaware of this website so I had to investigate!

Let's check it out!

RetroAchievements
RetroAchievements is an online service which provides users with fan-made achievement sets for many retro gaming platforms such as the Sega Master System. Navigate to RetroAchievements website. Choose Sign up Create an account. Download RALibretro emulator which facilitates achievements integration to retro games!

Launch RALibretro emulator | Choose RetroAchievements menu | Login | Enter RetroAchievements site info:

Cores
RetroAchievements cores are a special version of emulator cores. Essentially a core is a plug-in or emulator that runs platform specific games. Choose Settings menu | Manage Cores... | Master System and Download

Simpsons Trivia
Download Simpsons Trivia v1.02. Choose RALibretro File menu | Select Core | Master System | Genesis Plus GX ensures audio support. Choose File menu | Load Game | SimpsonsTrivia-v1.02.sms. Game ready to play:

IMPORTANT: Simpsons Trivia v1.02 must be selected as this has the correct Retro Achievements game hash! Choose arrow keys for Left, Right, Up, Down movement. Choose Z for Fire1, X for Fire2, P or Enter to Pause.

Implementation
Simpsons Trivia game has 4x difficulty categories: Easy, Normal, Hard and Pro! plus 4x questions rounds: 5, 10, 25, and 50. Thus 20x achievements setup: Perfect win for 4x categories * 4x rounds plus "Get between 80-100%" for 4x categories. Shout out to Bl4h8L4hBl4h setting up 20x achievements as per documentation!

Also, shout out to BenGhazi explaining how RALibretro emulator processes achievements e.g. Achievement has 4x important pieces of data stored in RAM: Difficulty [0x00], Selected Rounds [0x05], Correct Answers [0x05] and ScreenID [0x0C]. When all values are aligned on Game Over screen then the achievement is set!

IMPORTANT: here is Tag 1.02 source code excerpt which corroborates Game Over screen definition [0x0C]:

  // Screen type.
  #define SCREEN_TYPE_NONE	0
  #define SCREEN_TYPE_SPLASH	1
  #define SCREEN_TYPE_TITLE	2
  #define SCREEN_TYPE_INTRO	3
  #define SCREEN_TYPE_DIFF	4
  #define SCREEN_TYPE_LONG	5

  #define SCREEN_TYPE_READY	6
  #define SCREEN_TYPE_LEVEL	7
  #define SCREEN_TYPE_NUMBER	8
  #define SCREEN_TYPE_PLAY	9
  #define SCREEN_TYPE_QUIZ	10
  #define SCREEN_TYPE_SCORE	11
  #define SCREEN_TYPE_OVER	12

GitHub
RetroAchievements GitHub includes the following source code repositories for game achievement integration:

rcheevos	Library to parse and evaluate achievements and leaderboards for RetroAchievements
RAInterface	Enables RetroAchievements emulators to interact with the server via RA_Integration.dll
RAIntegration	The DLL responsible to integrate emulators with RetroAchievements.org
RALibretro	RALibretro is the multi-emulator used to develop RetroAchievements

Clone
On Windows, launch Terminal. Create C:\GitHub\RetroAchievements directory. Git clone the following repos:

 mkdir -p C:\GitHub\RetroAchievements
 cd C:\GitHub\RetroAchievements
 git clone --quiet https://github.com/RetroAchievements/rcheevos.git
 git clone --quiet https://github.com/RetroAchievements/RAInterface.git
 git clone --recursive --depth 1 -q https://github.com/RetroAchievements/RAIntegration.git 
 git clone --recursive --depth 1 -q https://github.com/RetroAchievements/RALibretro.git

IMPORTANT: use the --quiet [or -q] flag in order to suppress all the superfluous submodule logging output!

rcheevos
rcheevos is a C library that makes it easier to process Retro Achievements data. Launch Visual Studio 2022. Open rcheevos-test.sln. Choose Debug | x64. Rebuild Solution. Open test/test.c. Press F5 to Debug source:

RAInterface
RAInterface is a submodule which provides emulator hooks to integrate with RA server via RA_Integration.dll

RAIntegration
RAIntegration is the main DLL used for interfacing with retroachievements.org. Launch Visual Studio 2022. Open RA_Integration.sln. Choose Debug | x64. Rebuild Solution. Install CppUnitTest Test Adapter to run all Interface + Integration tests: Extensions menu | Manage Extensions | Test Adapter CppUnitTest Framework

Launch Test Explorer | Test menu | Test Explorer | Right click specific test e.g. RA_Interface.Tests | Debug:

RALibretro
RALibretro is the multi-emulator used to develop RetroAchievements and earn them. RALibretro uses libretro cores to do the actual emulation with RAIntegration DLL to connect with the site. Launch Visual Studio 2022. Open RALibretro.sln. Choose Debug | x64. Rebuild Solution. Open src/main.cpp. Press F5 to Debug source:

Overview
Here is an overview of the 1x C project rcheevos and 3x C++ projects to interface + integrate + emulate:

Project	Role	Dependencies
rcheevos	Core engine: parse + evaluate achievements / leaderboards	None
RAInterface	Interface library for emulators to interact with the RA stack	rcheevos
RAIntegration	Full DLL integration [login/server/achievement submission]	RAInterface + rcheevos
RALibretro	Multi-emulator that uses RA stack to support achievements	RAIntegration

libretro
libretro is an API designed for retro games and emulators to be compiled as DLLs which can be used in front ends like RALibretro that implement the libretro API. This is reminiscent of the OpenAI Retro work we did in 2024 which also uses libretro cores but exposes them in Python with Reinforcement Learning environments.

Similar to RetroAchievements OpenAI Retro uses its own Game Integration tool to inspect memory for points of interest e.g. starting state, reward function + done condition whereas RALibretro uses Memory Inspector:

Summary
To summarize, the RetroAchievements integration into Simpsons Trivia game has been awesome as gamers continue to play this game and top the achievements more than seven years after it was published! In fact, many hard core gamers have published YouTube videos showcasing their achievements for others to follow:

The next step would be reseach the documentation fully and create some RetroAchievements sets myself J

Pytest Setup Cheat Sheet

noreply@blogger.com (SteveProXNA) — Mon, 15 Sep 2025 08:00:00 +0000

In 2020, we checked out Python Setup Cheat Sheet as an interpreted high-level programming language with all code samples' unit tests using unittest package TestCase class. However, since then we have learned that pytest allows writing shorter more readable tests with less boilerplate. Plus we would like to include mocks!!

Let's check it out!

Frameworks
When developing code in Python there are typically five Top Python Testing Frameworks that are favorable:

NAME	MONIKER	DESCRIPTION
unittest	PyUnit	The default Python testing framework built-in with the Python Standard Library
pytest	Pytest	Popular testing frameworks known for simplicity, flexibility + powerful features
noseTest	Nose2	Enhanced unittest version offering additional plugins to support test execution
DocTest	DocTest	Python Standard Library module generates tests within source code DocString
Robot	Robot	Acceptance testing keyword-driven module that simplifies testcase automation

Here are some reasons why pytest currently seems to be the most popular Python unit test framework out:

  1. Simple and Readable Syntax
     You write plain Python functions instead of creating large verbose classes
     Assertions use plain assert statements which provide more detailed output

  2. Rich Plugin Ecosystem
     Plugins like pytest-mock, pytest-asyncio, pytest-cov, and more
     Easy to extend pytest plugins or write your own custom plugins

  3. Powerful Fixtures
     Allows for clean and re-usable setup and teardown using fixtures
     Supports various test level scopes, autouse, and parametrization

  4. Test Discovery
     Automatically discovers tests in files named test_*.py
     No need to manually register tests or use loader classes

  5. Great Reporting
     Colored output, diffs for failing assertions, and optional verbosity
     Integrates easily with tools like coverage, tox, and CI/CD systems

  6. Supports Complex Testing Needs
     Parameterized tests (@pytest.mark.parametrize)
     parallel test execution (pytest-xdist) + hooks

pytest

  pip install pytest

Setup
Depending on your stack here is some great documentation to setup pytest on PyCharm, VS Code or Poetry.

Configuration
In pytest, pytest.ini is the main configuration file used to customize and control pytest behavior across the unit test suite. pytest.ini hosts pytest options, test paths, plugin settings and markers to attach to the test functions to categorize, filter or modify their behavior. Here is a sample pytest.ini configuration file as base:

  [pytest]  
  addopts = -ra -q
  testpaths = tests
  markers =
      slow: marks tests as slow (deselect with '-m "not slow"')
      db: marks tests requiring database

Fixtures
Fixtures are methods in pytest that provide fixed baseline for tests to run. Fixtures can be used to setup all preconditions for tests, provide data, or perform teardown after tests finished via @pytest.fixture decorator.

Scope
Fixtures have scope: Function, Class, Module + Session which define how long fixture available during test:

SCOPE	DESCRIPTION
Function	Fixture created once per test function and destroyed at end of test function
Class	Fixture created once per test class and destroyed at the end of test class
Module	Fixture created once per test module and destroyed at end of test module
Session	Fixture created once per test session and destroyed at end of test session

conftest
In pytest, conftest.py file is used to share fixtures across multiple tests. All the fixtures in conftest.py will be automagically detected without needing to import. conftest: typically scoped at test root directory structure.

Dependencies
Dependency Injection: when fixtures are requested by other fixtures although this adds complexity to tests!

autouse
Simple trick to avoid defining fixture in each test: use the autouse=True flag to apply fixture to all tests.

yield
When you use yield in fixture function setup code executes before yield and teardown executes after yield:

  import pytest  
  @pytest.fixture
  def my_fixture(): 
      # setup code
      yield "fixture value"
      # teardown code

Arguments
Use pytest fixtures with arguments to write re-usable fixtures that can easily share across tests also known as Parameterized fixtures using @pytest.fixture(params=[0, 1, 2]) syntax. Note: these fixtures should not be confused with the @pytest.mark.parametrize decorator which can be used to specify inputs and outputs!

Factories
Factories, in the context of pytest fixtures, are functions that are used to create and return instances of objects that are needed to generate test data or objects with specific configuration in re-usable manner:

conftest.py

unittest.py

 @pytest.fixture
 def user_creds(): 
   def _user_creds(name: str, email: str):
     return {"name": name, "email": email}  
   return _user_creds

 def test_user_creds(user_creds):
   assert user_creds("John", "x@abc.com")=={  
     "name": "John",  
     "email": "x@abc.com",
   }

Best practices for organizing tests include: Organizing Tests by Testing Pyramid, Structure Should Mirror Application Code, Group or Organize Fixtures and Organize Tests Outside Application Code for scalability.

Mocking
Mocking is technique that allows you to isolate pieces of code being tested from its dependencies so the test can focus on the code under test in isolation. The unittest.mock package offers Mock and MagicMock objects:

Mock
A mock object simulates the behavior of the object it replaces by creating attributes and methods on-the-fly.

MagicMock
Subclass of Mock with default implementations for most magic methods (__len__, __getitem__, etc.). Useful when mocking objects that interact with Python's dunder methods that enable custom behaviors for common operations.

Patching
Patching is technique that temporarily replaces real objects in code with mock objects during test execution. Patching helps ensure external systems do not affect test outcomes thus tests are consistent and repeatable.

IMPORTANT - Mocks are NOT stubs!
When we combine @patch decorator with return_value or side_effect it is a stub but from the mock package!

METHOD	DESCRIPTION
return_value	Specify the single value of Mock object to be returned when method called
side_effect	Specify multiple values of Mock object to be returned when method called

Difference
In pytest, Mock and patch are both tools for simulating or replacing parts of your code during testing. Mock creates mock objects while patch temporarily replaces real objects with mocks during tests to isolate code:

Mock

patch

  from unittest.mock import Mock  
  
  mock_obj = Mock()
  mock_obj.some_method.return_value = 42 
  result = mock_obj.some_method()  
  assert result == 42

  from unittest.mock import patch
  
  def external_function(): 
      pass
  
  @patch('module_name.external_function')  
  def test_function(mock_external): 
      mock_external.return_value = "Mock data"
      result = external_function() 
      assert result == "Mock data"

IMPORTANT
When creating mocks it is critical to ensure mock objects accurately reflect objects they are replacing. Thus, it is best practice to use autospec=True to ensure mock objects respect function signatures being replaced!

Assertions
For completeness, here is list of assertion methods to verify method on mock object was called during tests:

METHOD	DESCRIPTION
assert_called	verify specific method on mock object has been called during a test
assert_called_once	verify specific method on mock object has been called only one time
assert_called_once_with	verify specific method on mock object called once with specific args
assert_called_with	verify every time method on mock object called with fixed arguments
assert_not_called	verify specific method on mock object was not called during the test
assert_has_calls	verify the order in which specific method on mock object was called
assert_any_call	verify specific method on mock object has been called at least once

Monkeypatch
Monkeypatching is technique used to modify code behavior at runtime especially where certain dependencies or settings make it challenging to isolate functionality for example environment variables or system paths:

app.py

test_app.py

  import os
  def get_app_mode() -> str:
      app_mode = os.getenv("APP_MODE") 
      return app_mode.lower()

  def test_get_app_mode(monkeypatch):
      """Test behavior when APP_MODE is set."""
      monkeypatch.setenv("APP_MODE", "Testing") 
      assert get_app_mode() == "testing"

pytest-mock
pytest-mock is pytest plugin built on top of unittest.mock that provides an easy-to-use mocker fixture that can be used to create mock objects and patch functions. When you use mocker.patch() method provided by pytest-mock default behavior is to replace the object with MagicMock() so pytest-mock uses MagicMock().

  pip install pytest-mock

app.py

  import requests
  from http import HTTPStatus
  
  def get_user_name(user_id: int) -> str:
      response = requests.get(f"https://api.example.com/users/{user_id}")
      return response.json()['name'] if response.status_code == HTTPStatus.OK else None

test_app.py

  from http import HTTPStatus
  from app import get_user_name
  
  def test_get_user_name(mocker):
      mock_response = mocker.Mock()
      mock_response.status_code = http.HTTPStatus.OK
      mock_response.json.return_value = {'name': 'Test'}
      mocker.patch('app.requests.get', return_value=mock_response)
      result = get_user_name(1)
      assert result == 'Test'

Legacy
In many legacy Python codebases you may detect references to Mock(), MagicMock() and @patch decorator from unittest.mock with pytest. Teams often keep the old style unless there compelling reason to refactor it.

Recommendation
However, here are some recommendations to prefer pytest-mock and mocker fixture for future unit testing:

  1. Prefer pytest-mock and the mocker fixture
     Cleaner syntax than unittest.mock.patch
     Automatically cleaned up after each test
     Plays well with other pytest fixtures
     Centralizes all patching into one fixture (mocker)

  2. Use monkeypatch for patching env vars, system paths and etc.
     Prefer monkeypatch for clarity and idiomatic pytest style
     e.g. os.environ, system paths, or patching open()

  3. Avoid @patch decorators unless migrating old tests
     Can be harder to read or stack with multiple patches
     Better to use mocker.patch() inline as cleaner syntax

  4. Use autospec=True when mocking complex or external APIs
     Ensure mocks behave like the real objects (catch bad call signatures)

  5. Use fixtures to share mocks across tests
     When you have mock used by multiple tests then define it as a fixture

tl;dr
Prefer pytest-mock (mocker fixture) for readability and less boilerplate. Import tools like MagicMock, Mock, call, ANY from unittest.mock when needed. Avoid @patch unless needed — inline mocker.patch() is usually cleaner. Keep everything in one style within a test module for consistency.

pytest-asyncio
Concurrency allows a program to efficiently execute its tasks asynchronously i.e. executing tasks while other tasks are waiting. pytest-asyncio simplifies handling event loops + managing async fixtures thru unit testing.

  pip install pytest-asyncio

app.py

test_app.py

  import asyncio
  
  
  async def fetch_data():
      # Simulate I/O operation.
      await asyncio.sleep(1)
      return {"status": "OK", "data": [42]}

  import pytest
  from app import fetch_data
  
  @pytest.mark.asyncio
  async def test_fetch_data():
      result = await fetch_data()
      assert result["status"] == "OK" 
      assert result["data"] == [42]

Consequently AsyncMock from unittest.mock allows you to mock asynchronous functions and/or coroutines.

CI/CD
GitHub Actions is feature-rich CI/CD platform and offers an easy and flexible way to automate your testing processes. GitHub Actions mainly consist of files called workflows. The workflow file contains job or several jobs that consist of sequence of steps. Here is sample YAML file that will trigger the workflow on git push:

~/.github/workflows/run_test.yml

  name: Run Unit Test via Pytest
  on: [push]
  jobs:
    build:
      runs-on: ubuntu-latest
      strategy:
        matrix:
          python-version: ["3.10"]
      steps:
        - uses: actions/checkout@v3
        - name: Set up Python ${{ matrix.python-version }}
          uses: actions/setup-python@v4
          with:
            python-version: ${{ matrix.python-version }}
        - name: Install dependencies
          run: |
            python -m pip install --upgrade pip
            if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
        - name: Lint with Ruff
          run: |
            pip install ruff
            ruff --format=github --target-version=py310 .
          continue-on-error: true
        - name: Test with pytest
          run: |
            coverage run -m pytest  -v -s
        - name: Generate Coverage Report
          run: |
            coverage report -m

Summary
To summarize, we have setup pytest for more robust unit testing with mocks and stubs via patching. Looking forward there are additional ways to improve unit test development experience with pytest as per the article:

  1. Use Markers To Prioritise Tests
     Organize tests in such a way that prioritizes key functionalities first
     Running tests with critical functionality first provide faster feedback

  2. Do More With Less (Parametrized Testing)
     Parametrized Testing allows you to test multiple scenarios in single test function
     Feed different parameters into same test logic covering more scenarios + less code

  3. Profiling Tests
     Identify the slow-running unit tests using the --durations=XXX flag
     Use the pytest-profiling plugin to generate tabular and heat graphs

  4. Run Tests In Parallel (Use pytest-xdist)
     Use the pytest-xdist plugin to distribute tests across multiple CPUs
     Tests run in parallel, use resources better, provide faster feedback!

Cloud CI-CD Cheat Sheet II

noreply@blogger.com (SteveProXNA) — Sun, 31 Aug 2025 08:00:00 +0000

In the previous post, we checked out Cloud CI/CD Cheat Sheet to transition from the 1990s to modern day CI/CD. Now lets integrate GitLab with GitFlow SDLC to demonstrate the Kubernetes CI/CD pipeline benefits.

Let's check it out!

GitLab CI/CD

• Create .gitlab-ci.yml at the root of project
• this is the driver file that co-ordinates stages:
• Build / Lint / Deploy

gitlab-ci.yml

Variables

• Generic Variables used in all environments and environment specific variables to build software
• Rules that can be used to automate deployments to "lower" environments vs. Manual deployments
• YAML that builds the Docker image and push image to container registry of the developer's choice
• YAML that has instructions on how to deploy latest built Docker image to Kubernetes cluster

environments.yml	deployment-rules.yml

Artefacts

• YAML files that contain Helm chart artefacts used like Deployment and Service YAML
• YAML files that contain Values to be injected including environment specific variables

deployment.yaml	service.yaml

NOTE: Hardcoded non-sensitive variables stored in Values YAML files including all environment variables:

Whereas sensitive information is stored in Kubernetes secret resources and injected at deployment time.

GitFlow SDLC
Development

• GitLab source code repo has main branch for all the Prod deployments
• GitLab source code repo has develop branch as the integration branch
• develop branch for feature development and deployment to DEV / UAT
• GitFlow: ensure develop branch is stable: cut feature branch off develop

Deployment

• Submit Pull Request | Merge to develop branch | Trigger build
• Auto-deploy to DEV | Manual deploy to UAT [when QA ready]

Testing

• Feature completed on DEV / preliminary testing on UAT cut release branch off develop
• Deploy release branch to UAT - complete feature testing and regression testing
• Any bugs on UAT in release candidate then cut bugfix branch off release branch
• Fix bug | Submit Pull Request | Merge to release branch | Re-deploy to UAT [manually]

Release

• Once release candidate is stable / all bugs fixed: then submit Pull Request release branch to main
• This action will build pipeline but NOT deploy!! Manually deploy to Prod when stakeholders agree!!

Alignment

• Finally, after deploy to Prod from main submit PR from main to develop for alignment
• Hotfixes available similar to bugfix | Cut hotfix branch from main and submit PR deploy to Prod
• After hotfix merged to main and deployed to Prod submit PR from main to develop for alignment

Kubernetes Management: Rancher
Q. What is Rancher?
Open source platform that simplifies the deployment, scaling and management of your Kubernetes clusters:

• Kubernetes: open source orchestration platform that automates management of containerized apps
• Rancher: open source container platform built on top of Kubernetes to simplify cluster management
• Download Kubernetes cluster configuration kubeconfig files from Rancher to connect to your clusters

Kubernetes kubeconfig

• kubeconfig file is YAML configuration used to connect to Kubernetes clusters, users and contexts
• Download DEV kubeconfig file from Rancher to localhost ~/.kube/dev-config
• Download UAT kubeconfig file from Rancher to localhost ~/.kube/uat-config

SETUP

  # Setup the global KUBECONFIG environment variable
  export KUBECONFIG=~/.kube/config:~/.kube/dev-config:~/.kube/uat-config
  # Flatten multiple kubeconfig files into one "master" kubeconfig file
  kubectl config view --flatten > one-config.yaml
  # Rename accordingly
  mv one-config.yaml ~/.kube/config
  # Confirm cluster configuration update
  kubectl config get-contexts

Deployment Verification
Monitor cluster - What is kubectl?

• Command line tool run commands against Kubernetes clusters - communicate using Kubernetes API
• Post-deployment use kubectl commands to verify the health of cluster ensuring all pods re-spawned

TEST Deployment
Finally, test endpoint(s) via curl or in Postman:

  # Test endpoint
  kubectl port-forward service/flask-api-service 8080:80
  curl http://localhost:8080/api/v1 --header "Content-Type: application/json"
  # RESPONSE
  {"message": "Hello World (Python)!"}

CI/CD Pipeline Benefits
Four Benefits of CI/CD - successful pipeline strategy helps your team deliver higher quality Software faster!

• Increased speed of innovation + automation = deployments that are faster and more regular
• Code in Production adds immediate value instead of sat waiting in a deployment queue!
• Engineers become more productive instead of focus on boring / mundane manual tasks
• Higher code quality due to continuous automated build, test, deploy rinse + repeat cycles J

Summary
To summarize, we have now highlighted the back story transitioning from the 1990s to modern day CI/CD and outlined the integration process with GitFlow SDLC to demonstrate Kubernetes CI/CD pipeline benefits!

Cloud CI-CD Cheat Sheet

noreply@blogger.com (SteveProXNA) — Thu, 31 Jul 2025 08:00:00 +0000

In 2024, we checked out GitLab Cheat Sheet to streamline collaborative workflow and then leverage CI/CD pipelines. However, it is interesting to tell the back story how we got from the 1990s to modern day CI/CD.

Let's check it out!

Evolution of SoftwareDeployment: Physical Servers to Container Orchestration

Era of Physical Servers: 1990s and Before
Back in the 1990s Software was predominantly deployed directly onto physical servers, many often housed in on-premises data centers. Each server typically dedicated to specific application [or set of applications].

Challenges: Scalability, Isolation, Resource Utilization

• involved procuring, setting up, deploying to additional physical servers = time consuming + expensive
• multiple apps could interfer with one another leading to system crashes or other performance issues
• some servers underutilized while others overwhelmed which meant inefficient resource distribution

Dawn of Virtualization: 2000s
Introduction of virtualization technologies like those provided by VMware allowed Virtual Machines [VMs] to each run a physical server which meant each VM operating as though it were on own dedicated hardware.

Benefits: Resource Efficiency, Isolation, Snapshot + Cloning

• multiple VMs could share resources of single server leading to better resource utilization
• VMs provide new level of isolation btwn apps = failure of one VM did not affect other VM
• VM state could be saved + cloned making it easier to replicate environments for scaling

Containerization: Rise of Docker
Next significant shift was containerization with Docker at the forefront. Unlike VMs, containers share host OS running in isolated User space which is lightweight and portable and can startup / shutdown more rapidly.

Advantages: Speed, Portability, Density

• containers start almost instantly i.e. applications launched and scaled in only a matter of seconds
• container images are consistent across environments = it works on my machine issues minimized
• lightweight nature = many containers run on host machine = better resource utilization than VMs

Container Orchestration: Enter Kubernetes
Increased container adoption prompted the need for container orchestration technologies like Kubernetes to manage scale and monitor containerized applications especially those hosted by managed Cloud providers.

Functions: Auto-scaling, Self-healing, Load Balancing, Service Discovery

• orchestration systems can automatically scale apps based on denamd or sudden traffic spikes
• if container or node fails then the orchestrator can restart or replace it = increased reliability!
• incoming requests are automatically distributed across containers ensure optimal performance
• as containers move across nodes, services can be discovered without any manual intervention

Summary of Definitions
Docker

• Platform as a Service product that uses OS-level virtualization Software in packages as containers
• Containers are isolated from one another bundle their own software, libraries, and configurations
• All containers share signle OS kernel on host thus use fewer resources than Virtual Machines

Kubernetes

• Open-source container orchestration system automating app deployment, scaling and management
• Runs containerized applications in cluster host machines from containers typically built using Docker

Helm

• Kubernetes package manager simplifies managing and deploying applications to clusters via "Charts"
• Helm facilitates configuration separated out in Values files and scaled out across all environments

Summary of Technology
Docker

• Dockerfile
• Image
• Container

text file that contains all commands used to assemble a Docker image template
executable package that includes code, runtime, environment variables and config
running instance of a Docker image isolated from other processes running on host

Kubernetes

• Namespace
• Workload
• Pod
• Node
• Replicaset
• Deployment
• Service

scope cluster resources and a way to isolate Kubernetes objects
containerized application running within the Kubernetes cluster
smallest deployable unit as created and managed in Kubernetes
workloads are placed in Containers on Pods to be run on Nodes
maintains a stable set of replica pods available running any time
provide a declarative way to update all Pods and Replicasets
abstract way to expose an application running on a set of Pods

DEMO Hello World

• Execute code on localhost [IDE]
• Build Docker image and locally
• Provision local Kubernetes cluster

TEST after deployment
curl http://localhost:8080
Hello World

Python Flask API application:

DEMO Docker Commands

  # Create KinD cluster
  kind create cluster --name flask-cluster
  # Create Dockerfile | Build Docker image
  docker buiild --pull -rm -f "Dockerfile" -t flask-api:latest "."
  # Execute Docker container
  docker run --rm -d -p 8080:8080/tcp flask-api:latest
  # Test endpoint
  curl http://localhost:8080

Dockerfile
KinD = Kubernetes in Docker is a tool for running local Kubernetes cluster using Docker container "nodes".

DEMO Kubernetes Commands

  # Load image into KinD cluster
  kind load docker-image flask-api:latest --name flask-cluster
  # Setup KinD cluster
  kubectl create ns test-ns
  kubectl config set-context --current --namespace=test-ns
  # Rollout Kubernetes Deployment and Service resources
  kubectl apply -f Kubernetes.yaml
  # Test endpoint
  curl http://localhost:8080

Kubernetes.yaml

LIMITATIONS
DEMO Hello World is sufficient to demonstrate the process on localhost but has many real world limitations!

Limitations

• Everything is on localhost - Cloud Computing typically requires Kubernetes cluster(s)
• Manually build Docker image from the Dockerfile
• Manually push Docker image to container registry
• Manually deploy running Docker container into Kubernetes cluster [Deployment exposed as Service]
• All Kubernetes resource values are hardcoded into declarative YAML file [Deployment and a Service]
• No facility to scale deployment across multiple environments: DEV, IQA, UAT, Prod
• Environment variables can be injected but is very brittle and cumbersome process
• No real immediate and secure way to inject secret information into deployment [secret password]

Solution
Next step is to integrate GitLab CI/CD pipeline to solve these issues and automate build deployment process!
This will be the topic of the next post.

Cloud Setup Cheat Sheet II

noreply@blogger.com (SteveProXNA) — Mon, 02 Jun 2025 08:30:00 +0000

In the previous post, we checked out Cloud Cheat Sheet to explain cluster provisioning process for managed cloud providers such as Azure AKS. Now we will resume to provision clusters: Amazon EKS and Google GKE.

Let's check it out!

Pre-Requisites
This blog post assumes an Azure, AWS, GCP account is setup plus all the corresponding CLIs are configured!

AWS Login
Navigate to https://aws.amazon.com | Sign In | Sign in using root user email. Root user | Root user email address e.g. steven_boland@hotmail.com | Next | Enter password. Setup AWS Multi-Factor Authentication.

AWS Single Sign On
Accessing AWS clusters programmatically is recommened to setup and configure AWS SSO. Example config:

  sso_start_url = https://stevepro.awsapps.com/start
  sso_region = eu-west-1
  sso_account_id = 4xxxxxxxxxx8
  sso_role_name = AdministratorAccess
  region = eu-west-1
  output = json

eksctl
Command-line tool that abstracts complexity involved in setting up AWS EKS clusters. Here is how to install:

Linux

curl --silent --location "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$(uname
-s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

Mac OS/X

brew tap eksctl-io/eksctl

brew install eksctl

Windows

Launch Powershell

choco install eksctl

Master Key
Next, create master SSH key for secure, automated and controlled access to your Kubernetes infrastructure:

cd ~/.ssh
ssh-keygen -t rsa -b 4096 -N '' -f master_ssh_key

eval $(ssh-agent -s)
ssh-add master_ssh_key

Amazon EKS
Amazon provides Elastic Kubernetes Service as a fully managed Kubernetes container orchestration service. Follow all instructions below in order to provision a Kubernetes cluster and test its functionality end-to-end. Download code sample here.

Pre-Requisites

  aws sso login

Check Resources

  aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --output table
  aws ec2 describe-addresses --query 'Addresses[*].PublicIp' --output table
  aws ec2 describe-key-pairs --query 'KeyPairs[*].KeyName' --output table
  aws ec2 describe-volumes --query 'Volumes[*].VolumeId' --output table
  aws ec2 describe-vpcs --query 'Vpcs[*].VpcId' --output table
  aws cloudformation list-stacks --query 'StackSummaries[*].StackName' --output table
  aws cloudwatch describe-alarms --query 'MetricAlarms[*].AlarmName' --output table
  aws ecr describe-repositories --query 'repositories[*].repositoryName' --output table
  aws ecs list-clusters --query 'clusterArns' --output table
  aws eks list-clusters --query 'clusters' --output table
  aws elasticbeanstalk describe-environments --query 'Environments[*].EnvironmentName' --output table
  aws elb describe-load-balancers --query 'LoadBalancerDescriptions[*].LoadBalancerName' --output table
  aws elbv2 describe-load-balancers --query 'LoadBalancers[*].LoadBalancerName' --output table
  aws iam list-roles --query 'Roles[*].RoleName' --output table
  aws iam list-users --query 'Users[*].UserName' --output table
  aws lambda list-functions --query 'Functions[*].FunctionName' --output table
  aws rds describe-db-instances --query 'DBInstances[*].DBInstanceIdentifier' --output table
  aws route53 list-hosted-zones --query 'HostedZones[*].Name' --output table
  aws s3 ls
  aws sns list-topics --query 'Topics[*].TopicArn' --output table
  aws sqs list-queues --query 'QueueUrls' --output table
  aws ssm describe-parameters --query 'Parameters[*].Name' --output table

Cluster YAML

  kind: ClusterConfig
  apiVersion: eksctl.io/v1alpha5
  
  metadata:
    name: stevepro-aws-eks
    region: eu-west-1
    version: "1.27"
    tags:
      createdBy: stevepro
  
  kubernetesNetworkConfig:
    ipFamily: IPv4
  
  iam:
    withOIDC: true
    serviceAccounts:
    - metadata:
        name: ebs-csi-controller-sa
        namespace: kube-system
      attachPolicyARNs:
      - "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
      roleOnly: true
      roleName: stevepro-aws-eks-AmazonEKS_EBS_CSI_DriverRole
  
  addons:
  - name: aws-ebs-csi-driver
    version: v1.38.1-eksbuild.2
    serviceAccountRoleARN: \
  	arn:aws:iam::4xxxxxxxxxx8:role/stevepro-aws-eks-AmazonEKS_EBS_CSI_DriverRole
  - name: vpc-cni
    version: v1.19.2-eksbuild.1
  - name: coredns
    version: v1.10.1-eksbuild.18
  - name: kube-proxy
    version: v1.27.16-eksbuild.14
  
  nodeGroups:
    - name: stevepro-aws-eks
      instanceType: m5.large
      desiredCapacity: 0
      minSize: 0
      maxSize: 3
      ssh:
        allow: true
        publicKeyPath: ~/.ssh/master_ssh_key.pub
      preBootstrapCommands:
        - "true"

Create Cluster

  eksctl create cluster -f ~/stevepro-awseks/cluster.yaml          \
     --kubeconfig ~/stevepro-awseks/kubeconfig                     \
     --verbose 5

Scale Nodegroup

  eksctl scale nodegroup                                           \
     --cluster=stevepro-aws-eks                                    \
     --name=stevepro-aws-eks                                       \
     --nodes=3                                                     \
     --nodes-min=0                                                 \
     --nodes-max=3                                                 \
     --verbose 5

Deploy Test

  kubectl create ns test-ns
  kubectl config set-context --current --namespace=test-ns
  kubectl apply -f Kubernetes.yaml
  kubectl port-forward service/flask-api-service 8080:80
  curl http://localhost:8080

Output

  Hello World (Python)!

Shell into Node

  kubectl get po -o wide
  cd ~/.ssh
  ssh -i master_ssh_key ec2-user@node-ip-address

Cleanup

  kubectl delete -f Kubernetes.yaml
  kubectl delete ns test-ns

Delete Cluster

  eksctl delete cluster                                            \
     --name=stevepro-aws-eks                                       \
     --region eu-west-1                                            \
     --force

  {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Allow",
             "Action": "*",
             "Resource": "*"
         }
     ]
  }

  {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Allow",
             "Action": ],
                 "ssm:GetParameter",
                 "ssm:GetParameters"
             ],
             "Resource": "arn:aws:ssm:*:*:parameter/aws/service/eks/optimized-ami/*"
         },
         {
             "Effect": "Allow",
             "Action": "ec2:DescribeImages",
             "Resource": "*"
         }
     ]
  }

Google GKE
Google provides the Google Kubernetes Engine as fully managed Kubernetes container orchestration service. Follow all instructions below in order to provision a Kubernetes cluster and test its functionality end-to-end.
Download code sample here.

Pre-Requisites

gcloud auth login
gcloud auth application-default login

gcloud auth configure-docker
gcloud config set project SteveProProject

Check Resources

  gcloud compute instances list
  gcloud compute disks list
  gcloud compute forwarding-rules list

  gcloud compute firewall-rules list
  gcloud compute addresses list
  gcloud container clusters list

Create Cluster

   gcloud container clusters create stevepro-gcp-gke               \
     --project=steveproproject                                     \
     --zone europe-west1-b                                         \
     --machine-type=e2-standard-2                                  \
     --disk-type pd-standard                                       \
     --cluster-version=1.30.10-gke.1070000                         \
     --num-nodes 3                                                 \
     --network=default                                             \
     --create-subnetwork=name=stevepro-gcp-gke-subnet,range=/28    \
     --enable-ip-alias                                             \
     --enable-intra-node-visibility                                \
     --logging=NONE                                                \
     --monitoring=NONE                                             \
     --enable-network-policy                                       \
     --labels=prefix=stevepro-gcp-gke,created-by=${USER}           \
     --no-enable-managed-prometheus                                \
     --quiet --verbosity debug

Get Credentials

  gcloud container clusters get-credentials stevepro-gcp-gke       \
     --zone=europe-west1-b                                         \
     --quiet --verbosity debug

IMPORTANT - if you do not have the following gke gcloud auth plugin then execute the following commands:

  gcloud components install gke-gcloud-auth-plugin
  gke-gcloud-auth-plugin --version

Deploy Test

  kubectl create ns test-ns
  kubectl config set-context --current --namespace=test-ns
  kubectl apply -f Kubernetes.yaml
  kubectl port-forward service/flask-api-service 8080:80
  curl http://localhost:8080

Output

  Hello World (Python)!

Shell into Node

  mkdir -p ~/GitHub/luksa
  cd ~/GitHub/luksa
  git clone https://github.com/luksa/kubectl-plugins.git
  cd kubectl-plugins
  chmod +x kubectl-ssh
  kubectl get nodes
  ./kubectl-ssh node gke-stevepro-gcp-gke-default-pool-0b4ca8ca-sjpj

Cleanup

  kubectl delete -f Kubernetes.yaml
  kubectl delete ns test-ns

Delete Cluster

  gcloud container clusters delete stevepro-gcp-gke                \
     --zone europe-west1-border                                    \
     --quiet --verbosity debug

Summary
To summarize, we have now setup and provisioned Azure AKS, Amazon EKS and Google GKE clusters with end-to-end tests. In future we could explore provisioning AWS and GCP Kubeadm clusters using Terraform!

Cloud Setup Cheat Sheet

noreply@blogger.com (SteveProXNA) — Mon, 05 May 2025 08:30:00 +0000

In 2024, we checked out GitLab Cheat Sheet to streamline collaborative team workflows that leverage CI/CD pipelines. Now, we will explain cluster provisioning process for managed cloud providers: Azure, AWS + GCP.

Let's check it out!

Pre-Requisites
This blog post assumes an Azure, AWS, GCP account is setup. The following links document paid or free tier:

Azure [Microsoft]	AZ	PAID Tier Account	FREE Tier Account
Amazon Web Services	AWS	PAID Tier Account	FREE Tier Account
Google Cloud Platform	GCP	PAID Tier Account	FREE Tier Account

Azure CLI
The Azure Command Line Interface is a set of commands used to create and manage Azure resources. The CLI is available across services designed to get working with Azure quickly with an emphasis on automation.

Linux
Install the Azure CLI on Linux | Choose an installation method e.g. apt (Ubunuty, Debian) | Launch Terminal

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

Mac OS/X
Install Azure CLI on Mac OS/X | Install with Homebrew | Install Homebrew manager if you haven't already!

brew update && brew install azure-cli

Windows
Install Azure CLI on Windows | Microsoft Install (MSI) | Download the Latest MSI of the Azure CLI (64-bit)

Download and install https://aka.ms/installazurecliwindowsx64

After installing the Azure CLI on Linux, Mac OS/X, Windows confirm the current working version of the CLI:

az version

AWS CLI
The AWS Command Line Interface is a unified tool used to manage your AWS services. Use the AWS CLI tool to download configure and control AWS services from the command line and automate them through scripts.

Linux
Install the AWS CLI on Linux | Linux tab | Command line installer - Linux x86 (64-bit) | Launch the Terminal

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Mac OS/X
Install the AWS CLI on MacOS/X | macOS tab | GUI installer | Download the macOS pkg file AWSCLIV2.pkg

Download and install https://awscli.amazonaws.com/AWSCLIV2.pkg

Windows
Install the AWS CLI on Windows | Windows tab | Download MSI | Download Windows (64-bit) AWSCLIV2.msi

Download and install https://awscli.amazonaws.com/AWSCLIV2.msi

After installing the AWS CLI on Linux, Mac OS/X, Windows confirm the current working version of the CLI:

aws --version

GCP CLI
The GCP Command Line Interface is used to create and manage Google Cloud resources + services directly from the command line and to perform common platform tasks faster by controlling cloud resources at scale.

Linux
Install the gcloud CLI | Linux tab | Platform Linux 64-bit (x86_64) | Launch Terminal + execute commands:

curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-linux-x86_64.tar.gz tar -xf google-cloud-cli-linux-x86_64.tar.gz
cd google-cloud-sdk	./install.sh

Mac OS/X
Install the gcloud CLI | macOS tab | Platform macOS macOS 64-bit (ARM64, Apple silicon) | Launch Terminal

curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-darwin-arm.tar.gz tar -xf google-cloud-cli-darwin-arm.tar.gz
cd google-cloud-sdk	./install.sh

Windows
Install the gcloud CLI | Windows tab | Download the Google Cloud CLI installer GoogleCloudSDKInstaller.exe

Download and install https://dl.google.com/dl/cloudsdk/channels/rapid/GoogleCloudSDKInstaller.exe

After installing the gcloud CLI on Linux, Mac OS/X, Windows confirm the current working version of the CLI:

gcloud init

gcloud version

Master Key
Next, create master SSH key for secure, automated and controlled access to your Kubernetes infrastructure:

cd ~/.ssh
ssh-keygen -t rsa -b 4096 -N '' -f master_ssh_key

eval $(ssh-agent -s)
ssh-add master_ssh_key

Azure AKS
Microsoft provides Azure Kubernetes Services as fully managed Kubernetes container orchestration service. Follow all instructions below in order to provision a Kubernetes cluster and end-to-end test its functionality.
Download code sample here.

Pre-Requisites

  az login

Check Resources

  az account list --output table
  az group list --output table
  az resource list --output table
  az resource list --query "[?location=='northeurope']" --output table
  az vm list --output table
  az aks list --output table
  az container list --output table
  az storage account list --output table
  az network public-ip list --output table

Create Group

  az group create --name stevepro-azraks-rg --location northeurope --debug

Security Principal

  az ad sp create-for-rbac --name ${USER}-sp --skip-assignment

Output

  {
     "appId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
     "displayName": "stevepro-sp",
     "password": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
     "tenant": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  }

Export

  export AZ_SP_ID=<value_from_appId>
  export AZ_SP_PASSWORD=<value_from_password>

Create Cluster

  az aks create --name stevepro-azraks                 \
     --resource-group stevepro-azraks-rg               \
     --dns-name-prefix stevepro-azraks                 \
     --node-count 3                                    \
     --node-vm-size Standard_D2s_v3                    \
     --kubernetes-version 1.31                         \
     --ssh-key-value ~/.ssh/master_ssh_key.pub         \
     --service-principal ${AZ_SP_ID}                   \
     --client-secret ${AZ_SP_PASSWORD}                 \
     --load-balancer-sku standard                      \
     --network-plugin azure --debug

Get Credentials

  export KUBECONFIG=~/.kube/config
  az aks get-credentials --name stevepro-azraks        \
     --resource-group stevepro-azraks-rg --file ~/.kube/config

Deploy Test

  kubectl create ns test-ns
  kubectl config set-context --current --namespace=test-ns
  kubectl apply -f Kubernetes.yaml
  kubectl port-forward service/flask-api-service 8080:80
  curl http://localhost:8080

Output

  Hello World (Python)!

Shell into Node

  mkdir -p ~/GitHub/luksa
  cd ~/GitHub/luksa
  git clone https://github.com/luksa/kubectl-plugins.git
  cd kubectl-plugins
  chmod +x kubectl-ssh
  kubectl get nodes
  ./kubectl-ssh node aks-nodepool1-20972701-vmss000000

Cleanup

  kubectl delete -f Kubernetes.yaml
  kubectl delete ns test-ns

Delete Cluster

  az aks delete --name stevepro-azraks                 \
     --resource-group stevepro-azraks-rg

Delete Group

  az group delete --name stevepro-azraks-rg --yes --no-wait
  az group delete --name NetworkWatcherRG --yes --no-wait

Summary
To summarize, we have setup CLIs for Azure, Amazon and Google and provisioned an Azure AKS Kubernetes cluster with end-to-end testing. Next, we will resume to provision clusters for Amazon EKS and Google GKE. This will be the topic of the next post.

Retrospective XVI

noreply@blogger.com (SteveProXNA) — Wed, 01 Jan 2025 10:00:00 +0000

Last year, I conducted a simple retrospective for 2023. Therefore, here is a retrospective for year 2024.

2024 Achievements

Transfer all Windows and Linux keyboard shortcuts and muscle memory to new Mac Book Pro
Transfer all important Windows and Linux applications navigations for M1-powered MacBooks
Build GitLab CI/CD pipelines extending DevOps skillset and streamline collaborative workflow
Provision Kubernetes clusters for GitLab CI/CD pipelines e.g. Azure AKS, AWS-EKS, GCP-GKE
Configure Doom open source port for Windows and Linux to debug step thru the source code
Launch fulltime Python coding experience to learn AI focusing on RL Reinforcement Learning
Experiment with OpenAI Gym project for RL research and build Atari available environments
Investigate OpenAI Retro project for RL research on classic Sega 8-bit + 16-bit video games

Note: building OpenAI projects for classic Sega 8/16-bit video games integration is a big achievement!

2025 Objectives

Document DevOps managed clusters provisioning experience with AWS / Azure / GCP providers
Channel cloud computing knowledge toward software architecture or infrastructure certification
Harness Python potential power invoking C/C++ [PyBind11] with code magnitudes times faster
Extend OpenAI Gym and Retro projects for more Indie video games + Reinforcement Learning!

Artificial Intelligence
Artificial Intelligence refers to capability of machines to imitate human intelligence. AI empowers machines to acquire knowledge, adapt and independently make decisions like teaching a computer to act human like.

Machine Learning
AI involves a crucial element known as Machine Learning. ML is akin to training computers to improve tasks without providing detailed instructions. Machines utilize data to learn and enhance the performance without explicit programming and concentrates on creating algorithms for computers to learn from data to improve.

Deep Learning
Deep Learning involves artificial neural networks inspired by the human brain: mimicking how human brains work. DL excels at handling complex tasks and large datasets efficiently and achieves remarkable success in areas like natural language processing and computer vision despite complexity and interpretation challenges.

Generative AI
Generative AI is the latest innovation in the AI field. Instead of just identifying patterns GenAI goes one step further by actually attempting to produce new content that closely resembles what humans might create.

Outline

Artificial Intelligence	Artificial Intelligence is the "big brain"
Machine Learning	Machine Learning is its learning process
Deep Learning	Deep Learning is its intricate wiring
Generative AI	Generative AI is the creative spark

Gen AI and LLMs are revolutionizing our personal and professional lives From supercharged digital assistants to seemingly omniscient chatbots these technologies are driving a new era of convenience, productivity, and connectivity.

Traditional AI uses predictive models to classify data, recognize patterns, + predict outcomes within specific context whereas Gen AI models generate entirely new outputs rather than simply making predictions based on prior experience.

This shift from prediction to creation opens up new realms of innovation: in healthcare traditional predictive model can spot suspicious lesion in lung tissue MRI whereas GenAI could also determine the likelihood that patient will develop pneumonia or other lung diseases and offer treatment recommendations based on best practices gleaned from thousands of similar cases.

Example
GenAI powered healthcare chatbots can assist patients and healthcare providers and medical administrators:

01. Symptom Checker	07. Mental Health Support
02. Appointment Scheduling	08. Insurance and Billing Assistance
03. Medication Reminders	09. Virtual Consultations and Telemedicine
04. Health Tips and Preventive Care	10. Health Records Access
05. Lab Results Interpretation	11. Emergency Triage
06. Chronic Disease Management

By leveraging conversational AI healthcare chatbot can improve patient engagement and provide real-time support and optimize the workflow for healthcare providers. Finally, Reinforcement Learning From Human Feedback RLHF can be integrated to further improve model performance over original pre-trained version!

Future
Artificial Intelligence is changing industries across the globe from healthcare and finance to marketing and logistics. As we enter 2025, the demand for skilled AI professionals continues to soar. Start out by building strong foundations in Python and understand key concepts such as machine learning and neural networks.

Therefore, whether an AI beginner or seasoned tech professional, here are the top 10 AI skills for success:

No.	AI Skill	Key Tools
01	Machine Learning (ML)	Scikit-learn, TensorFlow, PyTorch
02	Deep Learning	Keras, PyTorch, Google Colab
03	Natural Language Processing (NLP)	NLTK, SpaCy, GPT-based models (e.g., GPT-4)
04	Data Science and Analytics	NumPy, Pandas, Jupyter Notebooks
05	Computer Vision	OpenCV, YOLO (You Only Look Once), TensorFlow
06	AI Ethics and Bias Mitigation	AI Ethics Courses, Fairness Indicators (Google)
07	AI Infrastructure and Cloud Computing	Amazon Web Services, Microsoft Azure, Google Cloud AI
08	Reinforcement Learning	OpenAI Gym, TensorFlow Agents, Stable Baselines3
09	AI Operations (MLOps)	Docker, Kubernetes, Kubeflow, MLflow
10	Generative AI	Generative Adversarial Networks, DALL-E, GPT models

Finally, the GenAI market is poised to explode, growing to $1.3 trillion over the next 10 years from market size of just $40 billion in 2022. Therefore, it would be extraordinary to integrate GenAI to build content for OpenAI-based retro video games only to be trained by Reinforcement Learning algorithms to beat them J

OpenAI Retro Cheat Sheet

noreply@blogger.com (SteveProXNA) — Fri, 15 Nov 2024 09:00:00 +0000

In the previous post, we checked out OpenAI Gym: an open source library in Python used for reinforcement learning algorithms development to communicate between environments such as classic Atari video games.

In the this post, we check out OpenAI Retro: source code repository which lets you turn classic video games into Gym environments for reinforcement learning with classic Atari video games and many other Emulated Systems including: Nintendo GameBoy / GameBoy Color, the Sega MegaDrive and the Sega Master System.

Let's check it out!

Gym Retro
In May 2018, Open AI announce Gym Retro: a platform for reinforcement learning research on classic 8-bit and 16-bit video games. The release includes games from the Sega Genesis and Sega Master System, and Nintendo’s NES, SNES, and Game Boy consoles. The release also includes an Integration Tool that lets you create save states, find memory locations + design scenarios that reinforcement learning agents can solve.

Hello Gym Retro
Create new Hello Gym Retro type environment. Launch PyCharm | New Project | Name: HelloGymRetro [~/]

python -m venv .venv	pip install gym-retro
source .venv/bin/activate	pip install --upgrade pip

CODE

import retro
env = retro.make(game="Airstriker-Genesis")
observation = env.reset()

while True:
    action = env.action_space.sample()
    observation, reward, done, info = env.step(action)
    env.render()
    if done:
        observation = env.reset()
env.close()

IPMORTANT
At the time of this writing you may experience the following errors. Here are errors encountered + solutions:

ERROR #1	AttributeError: module 'gym.utils.seeding' has no attribute 'hash_seed'
Solution	pip uninstall -y gym	pip install gym==0.25.2

ERROR #2	ModuleNotFoundError: No module named 'gym.envs.classic_control.rendering'
Solution	pip uninstall -y gym	pip install gym==0.21.0

ERROR #3	ERROR: Could not find a version that satisfies the requirement gym==0.21.0
Solution	pip install git+https://github.com/openai/gym.git@9180d12e1b66e7e2a1a622614f787a6ec147ac40

Stable Retro
As per previous post, we learned that the lack of maintenance for OpenAI Gym prompted Gymnasium: which is an open source fork of the OpenAI Gym library maintained by The Farama Foundation. As Gym Retro built upon an unmaintained version of the OpenAI Gym library then this may explain errors encountered above J

Similarly, like Gym Retro built upon OpenAI Gym, Stable Retro is built upon Farama Foundation Gymnasium. Stable Retro lets you convert classic video games into Gymnasium environments for Reinforcement Learning!

Hello Stable Retro
Create Hello Stable Retro type environment. Launch PyCharm | New Project | Name: HelloStableRetro [~/]

python -m venv .venv	pip install stable-retro
source .venv/bin/activate	pip install --upgrade pip

CODE

import retro
env = retro.make(game="Airstriker-Genesis")
observation, info = env.reset()

while True:
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        observation, info = env.reset()
env.close()

Retro Examples
Complete all Retro examples! Extend OpenAI Test repo. Launch PyCharm | New Project | RetroCheatSheet

IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.

Setup ROMs: copy Sega ROM files from the data directory to Virtual Environment via the copy_files.py script

mkdir data
mkdir examples

cd examples
python copy_files.py

Sega Master System
Launch retro games for the Sega Master System 8-bit video game console with .sms supported ROM types:

pip install stable-retro	pip install --upgrade pip
AlienSyndrome-Sms	RType-Sms
BladeEagle-Sms	Shinobi-Sms
DickTracy-Sms	SonicTheHedgehog-Sms
JamesPond2CodenameRoboCod-Sms	TransBot-Sms

Sega Genesis
Launch retro games for the Sega Genesis [MD] 16-bit video game console with .md supported ROM types:

pip install stable-retro	pip install --upgrade pip
AlteredBeast-Genesis	SonicTheHedgehog2-Genesis
Flicky-Genesis	SonicTheHedgehog3-Genesis
GhoulsnGhosts-Genesis	StreetsOfRage2-Genesis
RevengeOfShinobi-Genesis	Strider-Genesis

YouTube

Pull Request
Finally, setup Pull Request for future contributions. Navigate to stable-retro source repo and Create new fork

Git clone the stable-retro forked source code repository and setup the following Git username and user email

git config --global --get user.name
git config --global --get user.email

git config --global user.name "SteveProXNA"
git config --global user.email "steven_boland@hotmail.com"

Next, configure the stable-retro upstream destination code repository in order to complete any Pull Request:

git remote -v	git remote add upstream git@github.com:Farama-Foundation/stable-retro.git
origin origin	git@github.com:SteveProXNA/stable-retro.git (fetch) git@github.com:SteveProXNA/stable-retro.git (push)
upstream upstream	git@github.com:Farama-Foundation/stable-retro.git (fetch) git@github.com:Farama-Foundation/stable-retro.git (push)

Launch PyCharm | Open Stable-Retro | Create virtual environment | Pip install all dependencies as required:

pip install -e .

pip install --upgrade pip

IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.

Examples
Navigate to the examples sub-directory. Checkout trivial_random_agent.py which should resemble the Hello Stable-Retro program above. Airstriker-Genesis ROM is included by default. Copy your Sega games into the corresponding data directory stable | game directory. Update name to the game ROM name. Execute script!

Tests
Navigate to the test_python sub-diretory. Open any test file | right click | Run test or Debug test to step thru source code. Important: you may need to install the pytest package if you get usual ModuleNotFound error:

pip install pytest

pip install --upgrade pip

Game Integration
As per the official Gym Retro docs Integrating a game means taking a video game ROM file and setting it up as a reinforcement learning environment by defining 3 things: #1 A starting state, #2 A reward function, #3 A done condition. However, in order to integrate a new game we must have the Game Integration tool setup.

Integration UI
Follow instructions as per video to install all dependencies then build Game Integration UI tool and launch:

sudo apt-get update
sudo apt-get install capnproto
sudo apt-get install libcapnp-dev
sudo apt-get install libqt5opengl5-dev
sudo apt-get install qtbase5-dev

sudo apt-get install zlib1g-dev
sudo apt autoremove
cmake . -DBUILD_UI=ON -UPYLIB_DIRECTORY
make -j$(grep -c ^processor /proc/cpuinfo)
./gym-retro-integration

Supported ROMs
ROM files contain the game itself. Each system has a unique file extension to denote the compatible system:

EXTN	SYSTEM	EXAMPLE
a26	Atari 2600	automaton.a26
gb	Nintendo Game Boy	dox-fire.gb
gba	Nintendo Game Boy Advance	Vantage-LostMarbles.gba
gg	Sega Game Gear	benryves-SegaTween.gg
md	Sega Genesis [Mega Drive]	Dekadence-Dekadrive.md
nes	Nintendo Entertainment System [Famicom]	Dr88-FamiconIntro.nes
pce	NEC TurboGrafx-16 [PC Engine]	chrisc-512_Colours.pce
sfc	Super Nintendo Entertainment System [Super Famicom]	Anthrox-SineDotDemo.sfc
sms	Sega Master System	blind-happy10.sms

Launch CLion | Open Stable-Retro. Navigate to src/ui folder. Open main.cpp. Click "Select CMakeLists.txt":

Navigate to CMakeLists.txt file | OK | Trust Project. Afterwards CLion will index the project and update all symbols. Click Edit configurations drop down list. Choose gym-retro-integration | Debug. Click Debug icon:

Finally, debug tests. Navigate to tests folder. Open data.cpp. Choose tests-data | Debug. Click Debug icon:

Process
Now that we have built the Game Integration tool setup either directly from the CLI and/or built from source it is now possible to integrate new games into Gym-Retro. This process involves: #1 Creating states for Gym Retro environment to load, #2 Inspect ROM memory for points of interest, #3 Find those address and types.

The following videos on OpenAI Game Integration tool Part 1, Part 2, Part 3 give some excellent examples on how to implement this process + create the following necessary files in order to Integrate brand new games:

Files

State file	Savestate from game beginning. Restart the environment puts agent at this point in the game
data.json	Defines the list of game-related variables that Python sees based on game memory addresses
scenario	JSON file defines the reward function and done condition using variables defined at data.json
metadata	JSON file defines starting state if no state specified by user and miscellaneous debugging info

Summary
To summarize, we have now setup Stable Retro environments for classic 8-bit video game systems like the Sega Master System and 16-bit video game systems like the Sega Genesis [MD] the next step would be to use reinforcement learning to train a bot to play classic emulated video games using this Gym Retro setup.

Consequently, Python NEAT is a popular library for training neural networks using genetic algorithms. There is an excellent Open AI and NEAT Tutorial series that integrates Sonic the Hedgehog as a testbed for applied reinforcement learning to provide a rich, rewarding, and visually complex environment that is well-suited for evolving neural networks. Ultimate goal: would be to use this as inspiration to integrate more Indie games!

OpenAI Farma Cheat Sheet

noreply@blogger.com (SteveProXNA) — Sun, 15 Sep 2024 09:00:00 +0000

OpenAI is an American Artificial Intelligence [AI] research organization founded Dec-2015 head-quartered in San Francisco. In April 2016, OpenAI released a public beta of "OpenAI Gym": its platform for reinforcement learning research. Reinforcement Learning [RL] is an interdisciplinary area of machine learning and optimal control concerned with how an agent takes action in environment in order to maximize cumulative rewards.

In 2021, the non-profit organization Farama Foundation took over Gym. They introduced new features into Gym and renamed the library to Gymnasium where the future maintenance of OpenAI Gym will take place.

Let's check it out!

Gym
OpenAI Gym is an open source Python library for developing reinforcement learning algorithms by providing standard API to communicate between algorithms and environments. Gym soon became widely adopted by the community for creating and training algorithms in various environments for AI Reinforcement Learning.

Gym provides a wide range of environments, including classic control problems, Atari games, and robotics. Gym also includes set of tools for monitoring and evaluating performance of reinforcement learning agents.

Hello Gym
Creating a Gym environment e.g. Cart Pole. Launch PyCharm | New Project | Name: HelloCartPoleGym [~/]

source .venv/bin/activate
pip install --upgrade pip

pip install gym
pip install gym[classic_control]

CODE

import gym
env = gym.make("CartPole-v1", render_mode="human")
observation, info = env.reset(seed=42)

for _ in range(1000):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        observation, info = env.reset()
env.close()

YouTube videos online also introduce OpenAI Gym but include Frozen Lake environment and Atari Breakout.

Gymnasium
The main problem with Gym was the lack of maintenance: OpenAI did not allocate substantial resources for the development of Gym since its inception seven years earlier and, by 2020, it simply was not maintained.

Gymnasium is a maintained fork of the OpenAI Gym library. The Gymnasium interface is simple, pythonic, and capable of representing general RL problems with a compatibility wrapper for old Gym environments:

Hello Gymnasium
Creating /Gymnasium environment e.g. Cart Pole. Launch PyCharm | New Project | HelloCartPoleGymnasium

source .venv/bin/activate
pip install --upgrade pip

pip install gymnasium
pip install gymnasium[classic-control]

CODE

import gymnasium as gym
env = gym.make("CartPole-v1", render_mode="human")
observation, info = env.reset(seed=42)

for _ in range(1000):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        observation, info = env.reset()
env.close()

Differences
While both Gym and Gymnasium are powerful tools for creating environments for reinforcement learning, it makes more sense to prefer Gymnasium going forward as technically future support for Gym there is none. Note: old import gym documentation is replaced with import gymnasium as gym convention going forward.

Farama Examples
Complete all Farama examples! Create OpenAI Test repo. Launch PyCharm | New Project | FarmaCheatSheet

IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.

Copy the official Gymnasium requirements.txt from the Farama Foundation GitHub repository docs directory:

pip install -r requirements.txt

pip install --upgrade pip

Classic Control
There are five Classic Control environments which are stochastic in terms of initial state within given range.

pip install gymnasium
Acrobot-v1	CartPole-v1	MountainCar-v0	Pendulum-v1

Toy Text
Toy Text environments are designed to be extremely simple, with small discrete state and action spaces, and hence easy to learn. They are suitable for debugging implementations of reinforcement learning algorithms.

pip install gymnasium
Blackjack-v1	CliffWalking-v0	FrozenLake-v1	Taxi-v3

Box2D
Box2D environments involve toy games based around box2d physics control and PyGame-based rendering.

pip install gymnasium		pip install Box2D
BipedalWalker-v3	CarRacing-v2	LunarLander-v2

MuJoCo
MuJoCo [Multi-Joint dynamics with Contact] is physics engine for facilitating research and development in robotics, biomechanics, graphics and animation, also areas where fast and accurate simulation is needed.

pip install gymnasium		pip install mujoco==2.3.0
Ant-v4	Humanoid-v4	InvertedPendulum-v4	Swimmer-v4
HalfCheetah-v4	HumanoidStandup-v4	Pusher-v4	Walker2d-v4
Hopper-v4	InvertedDoublePendulum-v4	Reacher-v4

Atari
Atari environments are simulated via the Arcade Learning Environment (ALE) through the Stella emulator.

pip install gymnasium[atari]==0.29.1		pip install gymnasium[accept-rom-license]
Adventure-v5	Defender-v5	KeystoneKapers-v5	Seaquest-v5
AirRaid-v5	DemonAttack-v5	KingKong-v5	SirLancelot-v5
Alien-v5	DonkeyKong-v5	Klax-v5	Skiing-v5
Amidar-v5	DoubleDunk-v5	Koolaid-v5	Solaris-v5
Assault-v5	Earthworld-v5	Krull-v5	SpaceInvaders-v5
Asterix-v5	ElevatorAction-v5	KungFuMaster-v5	SpaceWar-v5
Asteroids-v5	Enduro-v5	LaserGates-v5	StarGunner-v5
Atlantis-v5	Entombed-v5	LostLuggage-v5	Superman-v5
Atlantis2-v5	Et-v5	MarioBros-v5	Surround-v5
Backgammon-v5	FishingDerby-v5	MiniatureGolf-v5	Tennis-v5
BankHeist-v5	FlagCapture-v5	MontezumaRevenge-v5	Tetris-v5
BasicMath-v5	Freeway-v5	MrDo-v5	TicTacToe3D-v5
BattleZone-v5	Frogger-v5	MsPacman-v5	TimePilot-v5
BeamRider-v5	Frostbite-v5	NameThisGame-v5	Trondead-v5
Berzerk-v5	Galaxian-v5	Othello-v5	Turmoil-v5
Blackjack-v5	Gopher-v5	Pacman-v5	Tutankham-v5
Bowling-v5	Gravitar-v5	Phoenix-v5	UpNDown-v5
Boxing-v5	Hangman-v5	Pitfall-v5	Venture-v5
Breakout-v5	HauntedHouse-v5	Pitfall2-v5	VideoCheckers-v5
Carnival-v5	Hero-v5	Pong-v5	VideoChess-v5
Casino-v5	HumanCannonball-v5	Pooyan-v5	VideoCube-v5
Centipede-v5	IceHockey-v5	PrivateEye-v5	VideoPinball-v5
ChopperCommand-v5	Jamesbond-v5	Qbert-v5	WizardOfWor-v5
CrazyClimber-v5	JourneyEscape-v5	Riverraid-v5	WordZapper-v5
Crossbow-v5	Kaboom-v5	RoadRunner-v5	YarsRevenge-v5
Darkchambers-v5	Kangaroo-v5	Robotank-v5	Zaxxon-v5

IMPORTANT: when you pip install gymnasium[atari] without specifying tag 0.29.1 then you may receive gymnasium.error.NamespaceNotFound: Namespace ALE not found. Have you installed the proper package for ALE?

Pull Request
Finally, submit Pull Request: upload Farma examples. Navigate to source Gymnasium repo | Create new fork

Git clone the forked Gymnasium source code repository and setup the following Git username and useremail

git config --global --get user.name
git config --global --get user.email

git config --global user.name "SteveProXNA"
git config --global user.email "steven_boland@hotmail.com"

Next, configure the Gymnasium upstream destination code repository in order to complete the Pull Request:

git remote -v	git remote add upstream git@github.com:Farama-Foundation/Gymnasium.git
origin origin	git@github.com:SteveProXNA/Gymnasium.git (fetch) git@github.com:SteveProXNA/Gymnasium.git (push)
upstream upstream	git@github.com:Farama-Foundation/Gymnasium.git (fetch) git@github.com:Farama-Foundation/Gymnasium.git (push)

Launch PyCharm | Open Gymnasium | Create virtual environment | Install requirements as pre-requisites:

pip install -r docs/requirements.txt

pip install --upgrade pip

Uncomment game from ClassicControl sub-directory files.txt or ToyText files.txt. Open main.py. Run script:

Install Box2D and MuJoCo dependencies. Uncomment game from Box2D files.txt or MuJoCo. Run main.py:

Box2D
pip install Box2D

MuJoCo
pip install mujoco==2.3.0

Finally, in order to integrate Atari game examples I found this only worked cut from tag v0.29.1 [81b87ef]:

git fetch --tags --prune --prune-tags
git fetch --all

git tag | grep v0.29.1
git checkout tags/v0.29.1

It seems after tag v0.29.1 any Atari game may throw the following: gymnasium.error.NamespaceNotFound: Namespace ALE not found. Have you installed the proper package for ALE?. Resume Atari tag v0.29.1 setup:

pip install gymnasium[atari]

pip install gymnasium[accept-rom-license]

Retrospectives
In Jan-2020 Retrospective XI, we learned about an interesting development connecting Artificial Intelligence and retro video games leveraging Reinforcement Learning. In Jan-2021 Retrospective XII, we then discussed video games as an interesting field of study and retro game development of Pacman and Doom clones as an excellent avenue to pursue with regards to future applications in software with Artificial Intelligence and RL.

Consequently, in the previous post, we disovered how to debug step thru cloned versions of Doom to better understand the original source code: an important pre-requisite to extend knowledge for further research in reinforcement learning, in particular. For example, ViZDoom allows developing AI bots that play Doom using visual data. In 2022, ViZDoom joined Farama Foundation featuring new Gymnasium environment wrappers.

Summary
To summarize, we have now setup Gymnasium environments for Classic Control, Toy Text, Box 2D, MuJo Co and Atari. NB: all Atari environments leverage the Atari Arcade Learning Environment [ALE] emulator Stella.

In fact, reverting back to OpenAI which hosts the Retro Git source code repository, also including Emulated Systems like Atari. However, Retro also includes many other Emulated Systems we have developed for in the past including Nintendo GameBoy / GameBoy Color, Sega MegaDrive and or course the Sega Master System! This will be the topic of the next post.

Doom Code WalkThru

noreply@blogger.com (SteveProXNA) — Sat, 31 Aug 2024 10:30:00 +0000

Doom is a First-Person Shooter developed and published by id Software in December 1993. Doom is the first franchise installment and one of the most widely ported video games since the code was released Dec-1997.

Let's check it out!

Ports
In order to better understand the Doom source code we would like to debug step thru both the Doom port PrBoom on Linux and the Doom port DoomRetro on Windows written in C using modern-day software tools:

Linux	PrBoom	Cross-platform version of Doom ported to Linux
Windows	DoomRetro	Classic refined Doom source port for Windows PC

WAD Files
WAD is the acronym for "Where's All the Data?" which is the file format used by Doom and all Doom engine-based games for storing data. A WAD file consists of a header, a directory and the data lumps that make up the resources stored within the file. A WAD file can be one of two types: an Internal WAD or a Patch WAD:

IWAD	Internal WAD	Core WAD file loaded by engine that provides all data required to run game
PWAD	Patch WAD	Optional file that replaces IWAD data and provides additional data to engine

According to doomwiki there is List of notable WADs available to download beginning with Doom1.wad etc.

Doom Bible
The Doom Bible is the original design document which includes the specifications for the WAD file format structure that initiated an incredible modding scene in which fans created their own WAD files containing new levels commonplace for first-person shooter games! Now let's debug step thru Doom C source code.

PrBoom
PrBoom is a cross-platform version of the classic 3D first-person shooter Doom from id Software. PrBoom has since been ported to Linux and many other platforms. PrBoom requires the original IWAD data files.

Follow this YouTube video to setup PrBoom in VS Code and build and debug step through the source code! Launch Terminal. Install the following Linux packages as PrBoom pre-requisites, if not currently installed:

sudo apt-get install libghc-opengl-dev
sudo apt-get install libghc-sdl-dev

Download prboom-plus-2.5.1.3 from sourceforge.net. That is the link that seemed to work OK for me. Launch Terminal. Execute following commands to extract source code, configure, build and run PrBoom:

tar -xvf prboom-plus-2.5.1.3.tar.gz
cd ~/prboom-plus
chmod +x configure
./configure
make
chmod +x src/prboom-plus

If you get the following configure: error: *** You must have the SDL libraries installed before you can compile prboom *** See http://prboom.sourceforge.net/linux.html then install the following packages:

sudo apt-get install libsdl1.2-dev
sudo apt-get install libsdl-mixer1.2-dev
sudo apt-get install libsdl-net1.2-dev

Copy all files from the DoomCodeWalkThru Files directory to the local hidden directory ~/.prboom-plus

cp ~/GitHub/SteveProXNA/Doom/Files/* ~/.prboom-plus

Launch VS Code. Create build shell script, hidden .vscode directory with tasks.json and launch.json files

tasks.json

launch.json

 {
     "version": "2.0.0",
     "label": "build",
     "type": "shell",
     "linux": {
         "command": "./build.sh"
     },
 }

 {
   "version": "0.2.0",
   "configurations": [
     {
       "name": "(gdb) Launch",
       "type": "cppdbg",
       "request": "launch",
       "program": "${workspaceFolder}/src/prboom-plus",
       "args": [
         "-iwad",
         "Doom1.wad"
       ],
       "stopAtEntry": false,
       "cwd": "${workspaceFolder}",
       "externalConsole": false,
       "MIMode": "gdb",
     }
   ]
 }

Open i_main.c. Set breakpoints. Press Ctrl+Shift+B to build and F5 to debug step through source code!

IMPORTANT - the first character of IWAD file must be I not P otherwise may get an "IWAD not found" error.

CRITICAL - specify the arguments -iwad and WAD file e.g. Doom1.wad otherwise you will encounter error:

 M_LoadDefaults: Load system defaults.
  default file: ~/.prboom-plus/prboom-plus.cfg
   found ~/.prboom-plus/prboom-plus.wad
 
 prboom-plus v2.5.1.3 (http://prboom-plus.sourceforge.net/)
 I_SetAffinityMask: manual affinity mask is 1
 IdentifyVersion: IWAD not found

Keys

F1	Help
F2	Save game
F3	Load game
F4	Sound volume
F5	Change HUD
F6	Save game
F7	End game
F8	Message toggle
F9	Quick load
F10	Quit application
Escape	Menu back
Enter	Menu forward

Home	Restart game
Tab	Map
0-9	Change weapons
Left arrow	Rotate left
Right arrow	Rotate right
Up arrow	Move forward
Down arrow	Move back
Ctrl	Shoot
Space	Open door
<	Strafe left
>	Strafe right
/	Rotate 180

Note: F11 does Gamma correction. Also, re-copy Prboom-plus.cfg and Prboom-plus.wad to reset HUD.

The YouTube video shows some examples how to hack the d_englsh.h to make custom text and sounds!

DoomRetro
Doom Retro is the classic, refined Doom source port for Windows PC. Written in C, Doom Retro source code is maintained in doomretro Git repository and regularly compiled for both 32 and 64-bit using Visual Studio.

Launch Visual Studio 2022. Open doomretro.sln. Choose Debug | 32-bit. Right click Solution and Rebuild. Ensure the Platform Toolset is Visual Studio 2022 (v143) in the project Properties | Configuration Properties.

Open doomretro.c. Set breakpoints. Press Ctrl+Shift+B to build and F5 to debug step through source code! The function D_OpenWADLauncher() will prompt WAD file selection. Choose a WAD file from Files directory.

Keys

F1	Help
F2	Save game
F3	Load game
F4	Sound volume
F5	Graphics toggle
F6	Save game
F7	End game
F8	Message toggle
F9	Quick load
F10	Quit application
Escape	Menu back
Enter	Menu forward

Tab	Map
0-9	Change weapons
Left arrow	Rotate left
Right arrow	Rotate right
Up arrow [W]	Move forward
Down arrow [S]	Move back
Ctrl	Shoot
Space	Open door
A or <	Strafe left
D or >	Strafe right
Caps Lock	Always run toggle
Back Tick	Console commands

Commands
Press the back tick [`] key prompts the Console Commands dialog box. Here are some useful commands:

clear	Clears console
cmdlist	Lists all commands
god	Toggle God mode
health 100	Set health = 100%
playerstats	Shows player stats
quit	Quit application

maplist	List all maps to play
map E1M7	Warp to map E1M7
playerstats	Shows player stats
restartmap	Restart current map
newgame	Start new game
endgame	End the game

Documentation
The Game Engine Black Book: DOOM book is excellent reference material which goes into great depth about the development process of Doom in 1993. Masters of Doom documents the history of id Software and goes into great depth about development of many id Software franchises leading up-and-including Doom success.

Summary
To summarize, we have now setup Doom ports on both Linux and Windows computers to debug step through and better understand the source code. While this has been a fun and nostalgic exercise, many believe that this is crucial that Software Developers should review influential source code: rather than repeat the same past mistakes, we should study great works that preceded us and learn from and build upon their lessons.

Another motivation to study classic video game source code is their relevance to modern day AI topics such as Reinforcement Learning. For example: VizDoom allows developing AI bots to play Doom using only visual information. Hosted by The Farama-Foundation VizDoom allows creating environments based on the Doom engine. The library provides Python API wrappers for the OpenAI Gym / Gymnasium interface and provides an excellent opportunity to research machine visual learning and deep reinforcement learning in the future!

GitLab Cheat Sheet

noreply@blogger.com (SteveProXNA) — Thu, 04 Jul 2024 10:00:00 +0000

In 2016, we checked out GitHub Cheat Sheet: a distributed version control system for Open Source projects. Now we will explore differences in GitLab to streamline collaborative workflow and leverage CI/CD pipelines.

Let's check it out!

GitLab
As per previous post assumes an account setup on github.com. Create an account on gitlab.com if you have not already done so. As GitLab also uses underlying git we setup the following on localhost for development:

SSH Keys
An SSH key is an access credential for the SSH network protocol. Git supports both RSA and ED25519 keys. Launch Terminal. Issue the following commands to generate new SSH key and adding SSH key to ssh-agent:

RSA	ED25519
cd ~/.ssh	cd ~/.ssh
ssh-keygen -t rsa -b 4096 -C "steven_boland@hotmail.com"	ssh-keygen -t ed25519 -C <email>
Passphrase <redacted>	Passphrase <redacted>
eval "$(ssh-agent -s)"	eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa	ssh-add ~/.ssh/id_ed25519
xclip -selection clipboard < ~/.ssh/id_rsa.pub	xclip -selection clipboard < <key>

IMPORTANT:
If xclip is not installed then issue 2x commands: sudo apt-get update and sudo apt-get install xclip. Finally, navigate to github.com Settings | SSH and GPG keys | New SSH key | Paste contents from id_rsa.pub here.

Personal Access Token
If you favor HTTPS then Personal Access Tokens offer security benefits over password-based authentication. Launch gitlab.com. Navigate to Edit Profile | Access Tokens | Add new token. Create personal access token.

After new Personal Access Token is created upload into source control software for example Source Tree on Windows and Mac OS/X and JetBrains products on Linux that integrate Git like PyCharm for Python etc etc.

Source Tree
Launch Source Tree | Settings | Accounts Add... | Enter the following details | Paste personal access token:

JetBrains IDE
Launch PyCharm | File | Settings... | Version Control | GitLab. Click "+" button | Enter GitLab token | Log In:

Example
In 2021, we coded an example from scratch as full end-to-end Web API demo on local host, in Docker and Kubernetes. Let's replicate with Python Flask API but this time include Gitlab CI/CD deployment pipelines.

Navigate to gitlab.com. Create blank project. Launch PyCharm. Create the new project gitlabcheatsheet.

Enter main.py code. Install packages. Press F5 to debug. Launch Terminal | curl http://localhost:8080

main.py

Dockerfile

 from flask import Flask
 app = Flask(__name__)
 @app.route('/')
 def root():
     return "Hello World (Python)!\n"
 if __name__ == "__main__":
     app.run(host="0.0.0.0", port=8080)

 FROM python:3-alpine
 WORKDIR /service
 COPY requirements.txt .
 RUN pip install -r requirements.txt
 COPY . ./
 EXPOSE 8080
 ENTRYPOINT ["python3", "main.py"]

Code Dockefile. Launch Terminal | Enter the following Docker commands | curl http://localhost:8080

docker build --pull --rm -f "Dockerfile" -t flask-api:latest "."
docker run --rm -d -p 8080:8080/tcp flask-api:latest --name "flask-api"

Create local KinD cluster. Enter all the following Kubernetes Deployment and Service YAML configuration:

Kubernetes.yaml

 --- Deployment
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: flask-api-deployment
   namespace: test-ns
 spec:
   replicas: 1
   selector:
     matchLabels:
       app: flask-api
   template:
     metadata:
       labels:
         app: flask-api
     spec:
       containers:
         - name: flask-api
           image: flask-api:latest
           imagePullPolicy: Never
           ports:
             - containerPort: 8080

 --- Service
 apiVersion: v1
 kind: Service
 metadata:
   name: flask-api-service
   namespace: test-ns
 spec:
   selector:
     app: flask-api
   type: NodePort
   ports:
     - protocol: TCP
       port: 80
       targetPort: 8080

Finally test: Launch Terminal | Enter the following Kubernetes commands | curl http://localhost:8080

kubectl config set-context --current --namespace=test-ns
kind load docker-image flask-api:latest --name flask-cluster
kubectl apply -f Kubernetes.yaml

kubectl port-forward service/flask-api-service 8080:80
curl http://localhost:8080

Cluster
Extend example but this time include Gitlab CI/CD deployment pipelines. Ensure you have all Pre-Requisites:

Python 3.8
GitLab account [CI/CD]
Microsoft Azure subscription [AZ CLI]
VS Code Docker + Kubernetes

Docker Hub account
Kubernetes in Docker [KinD]
docker [CLI]
kubectl [CLI]

Azure AKS
Launch Terminal | Login to Azure portal. Enter commands to extract Service Principal and generate SSH key:

az login
az ad sp create-for-rbac --name ${USER}-sp
cd ~/.ssh

ssh-keygen -t rsa -b 4096 -N '' -f master_ssh_key
eval $(ssh-agent -s)
ssh-add master_ssh_key

Export the following environment variables. Note: use the Service Principal information for first 3x ENV VARs

export AZ_SP_ID=<value_from_appId>
export AZ_SP_PASSWORD=<value_from_password>
export AZ_TENANT_ID=<value_from_tenant>
export CLUSTER_NAME=stevepro-dev-cluster
export CLUSTER_NODES=3
export MASTER_SSH_KEY=~/.ssh/master_ssh_key.pub

export AZ_NETWORK_MODE=transparent
export AZ_CREATE_MODE=AKS
export AZ_LB_SKU=standard
export AZ_VM_SIZE=Standard_D2s_v3
export AZ_LOCATION=northeurope
export KUBERNETES_VERSION=1.28

Create the Azure resource group then finally create the AKS cluster. Note: this process can take few minutes

az group create --name ${CLUSTER_NAME} --location ${AZ_LOCATION} --debug

az aks create --name ${CLUSTER_NAME} --resource-group ${CLUSTER_NAME} \
--dns-name-prefix ${CLUSTER_NAME} --node-count ${CLUSTER_NODES} \
--node-vm-size ${AZ_VM_SIZE} --kubernetes-version ${KUBERNETES_VERSION} \
--ssh-key-value ${MASTER_SSH_KEY} --service-principal ${AZ_SP_ID} \
--client-secret ${AZ_SP_PASSWORD} --load-balancer-sku ${AZ_LB_SKU} \
--network-plugin azure --debug

Finally, execute following commands to download + export KUBECONFIG file on localhost and setup context

export KUBECONFIG=~/.kube/config
az aks get-credentials --name ${CLUSTER_NAME} \
--resource-group ${CLUSTER_NAME} --file ${KUBECONFIG}
kubectl create ns test-ns
kubectl config set-context --current --namespace=test-ns

Pipeline
Follow this tutorial to create and run your first GitLab CI/CD pipeline. Create .gitlab-ci.yml file at root the of the project directory. Refactor all YAML configuration in files beneath the cicd directory. Automate CI/CD to the Kubernetes cluster using helm charts. This way, multiple environments can be targeted and customized.

Variables
Navigate to gitlab.com | Settings | CI/CD | Variables. Enter all GitLab Variables here e.g. Docker registry host and username. Secure sensitive information like Docker password using Base64. Add KUBECONFIG:

Deployment
Trigger CI/CD pipeline. Launch Terminal | Repeat kubectl port-forward. Test: curl http://localhost:8080

kubectl port-forward service/flask-api-service 8080:80
curl http://localhost:8080

GitFlow
Gitflow is an alternative Git branching model that involves the use of feature branches and multiple primary branches. Here is practical guide to implement GitFlow using GitLab with the corresponding git commands.

Launch gitlab.com. Create new project. Default branch will be main. Git clone repository to localhost. For completeness cut initial Tag 0.1. In SourceTree setup GitFlow: Repository | Git-flow | Initialize Repository:

In gitlab.com set develop as default branch: GitLab | Setttings | Repository | Branch defaults | develop.

FEATURE [start]
git checkout -b feature/my-feature develop	# cut feature branch
git push --set-upstream origin feature/my-feature	# push feature to remote
git checkout develop
git merge --no-ff feature/my-feature	# merge feature to develop
git push	# push develop to remote
FEATURE [end]
git branch -d feature/my-feature	# delete local branch
git push origin -d feature/my-feature	# delete remote branch
RELEASE [start]
git checkout -b release/1.0 develop	# cut release branch
git push --set-upstream origin release/1.0	# push release to remote
BUGFIX [start]
git checkout -b bugfix/1.0 release/1.0	# cut bugfix branch
git push --set-upstream origin bugfix/1.0	# push bugfix to remote
git checkout release/1.0
git merge --no-ff bugfix/1.0	# merge bugfix to release
git push	# push release to remote
BUGFIX [end]
git branch -d bugfix/1.0	# delete local branch
git push origin -d bugfix/1.0	# delete remote branch
RELEASE [end]
git checkout main
git merge --no-ff release/1.0	# merge release to main
git push	# sync release with main
git tag -a 1.0 -m "Tag 1.0"	# cut tag 1.0 off main
git push origin tag 1.0	# push tag 1.0 to remote
git checkout develop
git merge --no-ff release/1.0	# merge release to develop
git push	# sync release with develop
git branch -d release/1.0	# delete local branch
git push origin -d release/1.0	# delete remote branch
HOTFIX [start]
git checkout main
git checkout -b hotfix/1.1 main	# cut hotfix branch
git push --set-upstream origin hotfix/1.1	# push hotfix to remote
HOTFIX [end]
git checkout main
git merge --no-ff hotfix/1.1	# merge hotfix to main
git push	# sync hotfix with main
git tag -a 1.1 -m "Tag 1.1"	# cut Tag 1.1 off main
git push origin tag 1.1	# push Tag 1.1 to remote
git checkout develop
git merge --no-ff hotfix/1.1	# merge hotfix to develop
git push	# sync hotfix with develop
git branch -d hotfix/1.1	# delete local branch
git push origin -d hotfix/1.1	# delete remote branch

IMPORTANT - you can compare two branches [source vs. target] anytime issuing the following commands:

git checkout develop	# source branch
git diff --name-only main	# target branch
OR	# target branch
git diff-tree --no-commit-id --name-only -r main..develop	# source vs. target branch

Summary
To summarize, now that we have discussed the differences between GitHub and GitLab, we have extended our previous end-to-end Web API demo on local host, in Docker and Kubernetes but now leveraged GitLab CI/CD pipelines to automate deployments to the cloud. Our next steps would be more GitFlow integration: Replicate deployments across multiple environments and complete the full Software Development LifeCyle!

MacOS Setup Cheat Sheet II

noreply@blogger.com (SteveProXNA) — Sat, 01 Jun 2024 10:00:00 +0000

In the previous post we checked out MacOS Setup Cheat Sheet to seamlessly transfer all keyboard shortcuts from Windows to Mac. Now we would like to transfer all Application navigation as M1 powered MacBooks are more appealing for companies than ever. Especially with the rise of A.I. requirements for faster data access.

Let's check it out!

Navigation
In the previous post, we began looking at using keyboard shortcuts on Mac. Now we would like to compare document shortcuts that we are used to for navigation on Windows however now using Ctrl instead of Cmd:

Home	Move to beg of line	Ctrl + Home	Move to beg of file
End	Move to end of line	Ctrl + End	Move to end of file
Ctrl + Left	Move to prev word	Ctrl + Shift + Left	Select the prev word
Ctrl + Right	Move to next word	Ctrl + Shift + Right	Select the next word
Shift + Home	Select to beg of line	Ctrl + Shift + Home	Select to beg of file
Shift + End	Select to end of line	Ctrl + Shift + End	Select to end of file

The main issue is there doesn't seem to be generic shortcut mapping; navigation seems application specific!

TextEdit
The Mac text system uses generalized key bindings mechanism that is completely re-mappable by all users. The standard bindings include all system defaults but you can override bindings in your own custom dict file:

SYSTEM	/System/Library/Frameworks/AppKit.framework/Resources/StandardKeyBinding.dict
CUSTOM	~/Library/KeyBindings/DefaultKeyBinding.dict

Here is how to override MacOS key bindings: Launch Terminal and create new a DefaultKeyBinding.dict file:

sudo bash
mkdir -p ~/Library/KeyBindings
touch ~/Library/KeyBindings/DefaultKeyBinding.dict
open -e ~/Library/KeyBindings/DefaultKeyBinding.dict

Typically, the default key bindings for the Home and End keys in MacOS are different to any other operating system. Thus you may like to fix the Home and End buttons first and in combination with Shift and Ctrl keys.

Next key bindings that may not be working is to move cursor from word to word. Integrate Keycode Modifier learnt from the previous post to complete custom dict file to change all MasOS key bindings. Here is my file:

{
	"\UF729" =	"moveToBeginningOfLine:";	/* Home Button*/
	"\UF72B" =	"moveToEndOfLine:";	/* End Button */
	"$\UF729" =	"moveToBeginningOfLineAndModifySelection:";	/* Shift+ Home Button */
	"$\UF72B" =	"moveToEndOfLineAndModifySelection:";	/* Shift+ End Button */
	"@\UF729" =	"moveToBeginningOfDocument:";	/* Ctrl + Home Button */
	"^\UF729" =	"moveToBeginningOfDocument:";	/* Alt + Home Button */
	"~\UF729" =	"moveToBeginningOfDocument:";	/* Win + Home Button */
	"@\UF72B" =	"moveToEndOfDocument:";	/* Ctrl + End Button */
	"^\UF72B" =	"moveToEndOfDocument:";	/* Alt + End Button */
	"~\UF72B" =	"moveToEndOfDocument:";	/* Win + End Button */
	"@$\UF729" =	"moveToBeginningOfDocumentAndModifySelection:";	/* Ctrl + Shift + Home Button */
	"^$\UF729" =	"moveToBeginningOfDocumentAndModifySelection:";	/* Alt + Shift + Home Button */
	"~$\UF729" =	"moveToBeginningOfDocumentAndModifySelection:";	/* Win + Shift + Home Button */
	"@$\UF72B" =	"moveToEndOfDocumentAndModifySelection:";	/* Ctrl + Shift + End Button */
	"^$\UF72B" =	"moveToEndOfDocumentAndModifySelection:";	/* Alt + Shift + End Button */
	"~$\UF72B" =	"moveToEndOfDocumentAndModifySelection:";	/* Win + Shift + End Button */
	"@\UF702" =	"moveWordLeft:";	/* Ctrl + Left */
	"^\UF702" =	"moveWordLeft:";	/* Alt + Left */
	"~\UF702" =	"moveWordLeft:";	/* Win + Left */
	"@\UF703" =	"moveWordRight:";	/* Ctrl + Right */
	"^\UF703" =	"moveWordRight:";	/* Alt + Right */
	"~\UF703" =	"moveWordRight:";	/* Win + Right */
	"@$\UF702" =	"moveWordLeftAndModifySelection:";	/* Ctrl + Shift + Left */
	"^$\UF702" =	"moveWordLeftAndModifySelection:";	/* Alt + Shift + Left */
	"~$\UF702" =	"moveWordLeftAndModifySelection:";	/* Win + Shift + Left */
	"@$\UF703" =	"moveWordRightAndModifySelection:";	/* Ctrl + Shift + Right */
	"^$\UF703" =	"moveWordRightAndModifySelection:";	/* Alt + Shift + Right */
	"~$\UF703" =	"moveWordRightAndModifySelection:";	/* Win + Shift + Right */
}

For completeness, here is a thorough KeyBindings tutorial explaining MacOS Keybinding Key Syntax and the Action Code. Detailed DefaultKeyBinding.dict examples shared online by users Belphemu, trusktr and zsimic.

Finally, in researching Default Key Bindings many posts promote Karabiner-Elements as a powerful keyboard customizer for macOS. However, it has been recommended not to use on M1 hardware due to kernel panics.

Terminal
Ideally we would like to update navigation on Terminal due to frequent use: Terminal | Settings... | Profiles. Keyboard tab customize Key and Action settings. Check Use Option as Meta Key to "backward delete word". Unfortunately, Terminal navigation not aligned therefore will prefer IDEs like VS Code or PyCharm terminals.

VS Code
Visual Studio Code is a cross-platform code editor available on Windows, Linux and Mac. However, unlike the TextEdit, VS Code does not respect DefaultKeyBinding.dict on MacOS. Therefore, change keyboard shorcuts!

Customize VS Code keybindings to align navigation: Launch VS Code | Settings | Keyboard Shortcuts | Add:

Command	Keybinding	Description
cursorTop	Ctrl + Home	Move to beg of file
cursorTopSelect	Ctrl + Shift + Home	Select to beg of file
cursorBottom	Ctrl + End	Move to end of file
cursorBottomSelect	Ctrl + Shift + End	Select to end of file
cursorWordLeft	Ctrl + Left	Move to prev word
cursorWordLeftSelect	Ctrl + Shift + Left	Select the prev word
cursorWordRight	Ctrl + Right	Move to next word
cursorWordRightSelect	Ctrl + Shift + Right	Select the next word
deleteWordLeft	Alt + Backspace	Delete the prev word

Continue to customize Menu keyboard shortcuts that one is accustomed to aligning VS Code navigation e.g.:

Command	Keybinding	Description
File: Open Folder...	Ctrl + K Ctrl + O	workbench.action.files.openFolderViaWorkspace
Terminal: Clear	Ctrl + L	workbench.action.terminal.clear
Terminal: Create New Terminal	Ctrl + Shift + T	workbench.action.terminal.new
renameFile	F2	renameFile

The keybindings for VS Code are updated in ~/Library/Application\ Support/Code/User/keybindings.json file. Update keybindings.json directly for Terminal keyboard shortcuts like delete word prior + cursor movement:

{
	"key":	"alt+backspace",
	"command":	"workbench.action.terminal.sendSequence",
	"args":	{ "text": "\u001b\u007f" },
	"when":	"terminalFocus"
},
{
	"key":	"cmd+backspace",
	"command":	"workbench.action.terminal.sendSequence",
	"args":	{ "text": "\u001b\u007f" },
	"when":	"terminalFocus"
},
{
	"key":	"cmd+left",
	"command":	"workbench.action.terminal.sendSequence",
	"args":	{ "text": "\u001bb" },
	"when":	"terminalFocus"
},
{
	"key":	"cmd+right",
	"command":	"workbench.action.terminal.sendSequence",
	"args":	{ "text": "\u001bf" },
	"when":	"terminalFocus"
},
{
	"key":	"cmd+z",
	"command":	"workbench.action.terminal.sendSequence",
	"args":	{ "text": "\u0003" },
	"when":	"terminalFocus"
},
{
	"key":	"cmd+f5",
	"command":	"workbench.action.debug.start",
},
{
	"key":	"f5",
	"command":	"workbench.action.debug.continue",
}

PyCharm Community
Align PyCharm Keymap to VS Code Keymap for consistent keyboard shortcuts and developer IDE experience. In the "Welcome to PyCharm" popup choose Plugins | VSCode Keymap | Install. Navigate back to Customize. Under Keymap change default value VSCode (macOS). Now most VS Code shortcuts will work like Windows.

However, in order to better align full navigation, update the following from PyCharm | Settings... | Keymap:

Editor Actions	Move Caret to Text Start	Ctrl + Home
Editor Actions	Move Caret to Text Start with Selection	Ctrl + Shift + Home
Editor Actions	Move Caret to Text End	Ctrl + End
Editor Actions	Move Caret to Text End with Selection	Ctrl + Shift + End
Editor Actions	Move Caret to Previous Word	Ctrl + Left
Editor Actions	Move Caret to Previous Word with Selection	Ctrl + Shift + Left
Editor Actions	Move Caret to Next Word	Ctrl + Right
Editor Actions	Move Caret to Next Word with Selection	Ctrl + Shift + Right
Editor Actions	Move Up and Scroll with Selection	Ctrl + Shift + Up
Editor Actions	Move Down and Scroll with Selection	Ctrl + Shift + Down
Editor Actions	Delete to Word Start	Alt + Backspace
Main Menu	Navigate Back	Alt + Up
Main Menu	Navigate Forward	Alt + Down
Main Menu	Navigate \| Go to Line: Column	Ctrl + G
Main Menu	Run \| Run/Debug \| Run	Ctrl + F5
Plugins	Terminal \| Clear Terminal	Ctrl + L

IMPORTANT: typically use Alt + Left to Navigate Back but this is now doubled to Move Word left as priority!

NOTE
If Ctrl + Shift + O launches Symbols then remove duplicate: Choose Settings | Keymap | filter as "symbol". Double click Navigate | Go to Symbol | Edit | Remove Ctrl+Shift+O | Apply | OK. Repeat this as necessary.

Excel
Keyboard shortcuts in Excel like Ctrl+Home and Ctrl+End won't work so prefer to use arrow keys to navigate

Ctrl + Left	Move to first cell of selected row
Ctrl + Right	Move to last cell of selected row
Ctrl + Up	Move to first cell of selected column
Ctrl + Down	Move to last cell of selected column

PostgreSQL
PostgreSQL is preferred by companies because it offers data types not found in MySQL. Plus MySQL can be used for web-based/online or mapping functions, whereas PostgreSQL is recommended for large analytical processes.

Postgres
Setup a local PostgreSQL database server on Mac OS/X. Download Postgres.app PostgreSQL 16 (Universal).

pgAdmin
Download and install local pgAdmin database client on Mac OS/X to administer and PostgreSQL instances.

PyCharm Professional
After installing Postgres server one can use pgAdmin client for data access. However, it seems challenging to customize navigation here. An excellent alternative is Database Tools + SQL plugin available from PyCharm.

This plugin is available only in PyCharm Professional but may prove to be a valuable investment longterm J

PyCharm Community	PyCharm Professional

Download and install PyCharm Professional for macOS. Launch PyCharm Professional. Import Settings from PyCharm Community Edition. Thus you should have all navigation aligned! Note the Database Tools + SQL plugin will be enabled by default. Interesting plugins PostgreSQL Debugger and SQL Optimizer + Indexing.

Connect to database: View menu | Database | Click "+" | Data Source | PostgreSQL. Enter credentials | OK.

Finally in Settings | Editor | General | Code Completion enable the following options: Automatically insert single suggestions, Sort suggestions alphabetically, Show suggestions as you type, Match case: First letter.

Summary
To summarize, armed with this knowledge we have setup the MacOS to better align keyboard shortcuts and application navigation techniques accumulated from years of Windows development experience. From here, we are in now an excellent position to leverage the M1 chip for faster data access as per A.I. requirements!

MacOS Setup Cheat Sheet

noreply@blogger.com (SteveProXNA) — Wed, 01 May 2024 08:00:00 +0000

In late 2020, Apple unleashed its industry-leading performant M1 chip designed specifically for the Mac. As a result, M1 powered MacBooks have provened to be much faster than Windows thus making Apple computers more appealing for companies than ever. Especially with the rise of A.I. requirements for faster data access.

The issue now for Windows users is to seamlessly transfer all keyboard shortcuts + muscle memory to Mac!

Let's check it out!

Documentation
If you are new to the Mac then the first thing to do is acquaint yourself with keyboard shortcuts as outlined in various online documentation pages and especially with what are those symbols shown in menus on Mac:

Keyboards
The major differences between Windows and Mac keyboards are the special keys found on either side of the spacebar known as modifier keys and are typically used for System Keybindings. Here is quick comparison:

Location	Windows	Mac OS	Keycode Modifier	Menu Symbol
Left	Ctrl	Ctrl	@	⌃
Middle	WinKey	Option	~	⌥
Right	Alt	Command	^	⌘
Left	Shift	Shift	$	⇧

Out-of-the-box, there are some common keyboard shortcuts shared across Windows and Mac, for example:

Alt + Tab	Toggle between all open applications
Ctrl + Tab	Toggle between all open tabs within an application

However, for all common Ctrl + type Windows keyboard shortcuts the Mac prefers to use the Command key:

Ctrl + A	Select all	Ctrl + Q	Quit application
Ctrl + C	Copy item	Ctrl + S	Save item
Ctrl + F	Find item	Ctrl + V	Paste item
Ctrl + N	New item	Ctrl + W	Close window
Ctrl + O	Open item	Ctrl + X	Cut item
Ctrl + P	Print item	Ctrl + Z	Undo action

Therefore, remap the modifier keys in System Settings | Keyboard | Keyboard Shortcuts... | Modifier Keys:

IMPORTANT: in order to retain Alt + Tab functionality we have now had to re-map Option to Command also! There is an AltTab utility but companies may disallow the installation of third party software, unfortunately.

System Settings
Customize your Mac with System Settings. Launch System Settings from Dock and make following changes:

General	Language & Region	Region = New Zealand
Accessibility	Keyboard	Full Keyboard Access
Accessibility	Pointer Control \| Trackpad Options	Use trackpad for scrolling = DISABLED
Appearance	Show scroll bars	Automatically based on mouse or trackpad
Desktop & Dock	Dock	Size = Small
Desktop & Dock	Dock	Position on screen = Left
Desktop & Dock	Desktop & Stage Manager	Click wallpaper to reveal desktop = Only in Stage Manager
Desktop & Dock	Group windows by application	DISABLED
Desktop & Dock	Displays have separate Spaces	DISABLED
Displays	Dual Monitors	1920 x 1080 (Default)
Users & Groups	Administrator	Update login photo
Keyboard	Key repeat rate	Fast
Keyboard	Delay until repeat	Short
Keyboard	Turn keyboard backlight off after inactivity	Never
Keyboard	Press fn key to	Do Nothing
Keyboard	Keyboard navigation	Enabled
Trackpad	Scroll & Zoom	Natural scrolling = DISABLED

Here is some more information explaining the rationale behind some of these System Settings options set:

Use trackpad for scrolling enables scroll bars visible unconditionally + not autohide
Displays have separate Spaces on dual monitors prevents other screen to go black
Displays | Dual Monitors | Ensure resolution is the same default for both monitors
Keyboard | Key repeat rate | Fast ensures that holding key down prevents accents
Trackpad | Natural scrolling Disabled ensures vertical scrolling naturally downward

Keyboard Shortcuts
Customize your general Mac keyboard shortcuts. Launch System Settings | Keyboard | Keyboard Shortcuts:

Here is my quick summary of Keyboard Shortcuts setup that will be compatible with Application navigation:

Launchpad & Dock	Turn Dock hiding on/off	DISABLED
Launchpad & Dock	Show Lauchpad	DISABLED
Mission Control	Show Desktop [F11]	DISABLED
Mission Control	Move left a space [Cmd + left]	DISABLED
Mission Control	Move right a space [Cmd + right]	DISABLED
Keyboard	Move focus to the window toolbar [Cmd + F5]	DISABLED
Keyboard	Move focust to next window [Ctrl + `]	ENABLED
Input Sources	Select previous input source [Cmd + Space]	DISABLED
Screenshots	Save picture of screen as a file [Ctrl + 3]	ENABLED
Screenshots	Save picture of selected area as a file [Ctrl + Shift + 4]	ENABLED
Spotlight	Show Spotlight search [Ctrl + Space]	DISABLED
Accessibility	Turn VoiceOver on or off [Ctrl + F5]	DISABLED
App Shortcuts	All Applications \| Rename	F2

Here is more information explaining the rationale behind some of these Keyboard Shortcuts... options set:

Disable F11 prevents the Desktop from always showing
Disable Cmd + left prevents move former word collision
Disable Cmd + right prevents move next word collision
Disable Cmd + F5 key prevents IDE debugging collision
Disable Cmd + Space prevents IDE intellisense collision
Disable Ctrl + Space prevents IDE intellisense collision
Disable Ctrl + F5 key prevents Voice Over alternating

Finder
The Finder is the home base for your Mac: you use the Finder to organize your folders and files on the Mac. Launch Finder | Settings... Leave all options in Finder Settings but check all checkboxes on Advanced tab:

In View menu check "as Columns" + "Sort By Name". Show all relevant Sidebar, Toolbar, Status options etc:

View menu | Select the following:

Check "as Columns"
Sort By | Name
Show Sidebar
Show Preview
Show Toolbar
Hide All Tabs
Hide Tab Bar
Show Path Bar
Show Status Bar
Hide View Options
Hide Preview Options

Here are some Windows keyboard shortcut equivalents. *Customization via App Shortcuts as shown below:

Ctrl + N	New Finder Window
Ctrl + Shift + N	New Folder
Ctrl + O	Open File
Ctrl + Shift + G	Go to Folder...
F2* \| Enter	Rename File / Folder
Alt + Backspace	Delete File / Folder

Shortcuts: Right click folder | Show in Enclosing folder. Right click folder the Status Bar | Copy as Pathname.

Finally, here are two handy aliases to hide and show private files and folders on Mac via ~/.bash_profile:

alias showFiles='defaults write com.apple.finder AppleShowAllFiles YES; killall Finder /System/Library/CoreServices/Finder.app'

alias hideFiles='defaults write com.apple.finder AppleShowAllFiles NO; killall Finder /System/Library/CoreServices/Finder.app'

Python
As per this post, Python is installed on most Mac OS/X systems by default. Also, one can install Python when you install Xcode Command Line Tools. Another option is update Python from the official website e.g. v3.12. IMPORTANT: when coding in Python activate the created virtual environment: source .venv/bin/activate.

As per this post, we installed open source Python distribution Anaconda used for data science. However, on the Mac after launch Terminal and enter jupyter notebook you may encounter the following couple errors:

Error #1: Jupyter Notebook 500 : Internal Server Error - follow instructions here to resolve the issue:

pip install --upgrade jupyterhub
pip install --upgrade --user nbconvert

Error #2: ModuleNotFoundError: No module named 'jupyter_server.contents' - follow instructions here:

pip uninstall traitlets
pip install traitlets==5.9.0
jupyter notebook
http://localhost:8889/notebooks

Terminal
The Mac terminal may be zsh by default. However, change your default shell via Settings... | General | Shell open with: Command path /bin/bash or enter chsh -s /bin/bash. Confirm BASH is now in Terminal title!

IMPORTANT: remove silence deprecation warning | Enter the following into Terminal for ~/.bash_profile:

echo "export BASH_SILENCE_DEPRECATION_WARNING=1" >> ~/.bash_profile

Customize the Mac terminal to always show full directory path as the cursor similar to the Linux terminal:

echo "export PS1='\u@\h:\w$ '" >> ~/.bash_profile

Finally, keyboard shortcuts like Ctrl + C or Ctrl + Z may not work to break on Mac thus prefer Ctrl + ".".

Homebrew
Homebrew is a package manager for macOS which lets you install free and open-source software using your terminal. Launch terminal and issue the following commands to install Homebrew and update bash_profile:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.bash_profile

Install free and open-source software for Cloud Computing for example: minikube, KinD, Helm, Terraform:

minikube	brew install minikube
KinD	brew install kind
Helm	brew install helm
Terraform	brew tap hashicorp/tap \| brew install hashicorp/tap/terraform

Cloud CLIs
Download and install Command Line Interfaces for all managed cloud providers e.g.: AWS, Azure, Google: Install the latest version of the AWS CLI. Download the macOS pkg file. Confirm installation aws version. Install Azure CLI on macOS using brew brew update && brew install azure-cli. Confirm install az version. Install the Google Cloud CLI. Download the Apple M1 package. Extract and execute install.sh | gcloud init.

Kubernetes
As per this post, install Kubernetes with Docker Desktop on Mac OS/X. Next, install and setup kubectl on the Mac OS/X. However, the default kubectl autocomplete setup quick reference may produce the following error

bash: completion: function `__start_kubectl` not found

If you experience this error then here is an alternative approach: first brew install bash bash-completion Set Terminal to use bash: Settings... | General | Shell open with: Command path /opt/homebrew/bin/bash

Finally, update ~/.bash_profile

alias k="kubectl"
complete -F __start_kubectl k
source /opt/homebrew/etc/profile.d/bash_completion.sh
source <(kubectl completion bash)

BASH Profile
Dump out the contents of the ~/.bash_profile file and verify all the following aliases and commands exist:

export BASH_SILENCE_DEPRECATION_WARNING=1	# https://support.apple.com/en-us/102360
export PS1='\u@\h:\w$ '	# Show always fullpath on terminal
eval "$(/opt/homebrew/bin/brew shellenv)"	# brew
alias d='docker'	# Docker
alias k='kubectl'	# Kubernetes
complete -F __start_kubectl k	# Kubernetes autocomplete
source /opt/homebrew/etc/profile.d/bash_completion.sh	# Kubernetes autocomplete
source <(kubectl completion bash)	# Kubernetes autocomplete
alias sba='source .venv/bin/activate'	# Python virtual environment
alias showFiles='defaults write com.apple.finder AppleShowAllFiles YES; killall Finder /System/Library/CoreServices/Finder.app'
alias hideFiles='defaults write com.apple.finder AppleShowAllFiles NO; killall Finder /System/Library/CoreServices/Finder.app'

Summary
To summarize, by remapping the modifier keys in System Settings has helped tremendously, however this method does not provide a complete one-to-one mapping for all keyboard shortcuts from Windows to Mac.

Another option is investigate global hotkeys in App Shortcuts for All Applications and specific Applications.

However, this could be extremely time-consuming and unsuccessful thus will keep the setup as is for now. Therefore, he is short list of mismatched keyboard shortcuts going forward and new key combos to learn:

General	Ctrl + H	Hide current window of this front app
General	Ctrl + M	Minimize all open windows to the Dock
General	Ctrl + 1	Restoring minimized window from Dock
General	Ctrl + `	Move focus to next window of same app
General	Win + Tab	Use instead of default Ctrl + Tab option
Applications	Ctrl + Enter	Save current open document then exit

After aligning keyboard shortcuts from Windows to Mac the next step is to transfer Application navigation! This will be the topic of the next post.

Retrospective XV

noreply@blogger.com (SteveProXNA) — Mon, 01 Jan 2024 09:00:00 +0000

Last year, I conducted a simple retrospective for 2022. Therefore, here is a retrospective for 2023.

2023 Achievements

Complete fifteen year blog on agile software application to game development and beyond
Enter Lil Evel Knievel in SMS Power! 2023 competition as side scrolling Wonderboy tribute
Port Sega Master System games to Sega MegaDrive then document development process
Apply Sega Master System development techniques to eBPF setup using C code examples
Resurrect test driven development and mock integration into Linux / C environment setup
Transition 8-bit Sega Master System development setup to the Nintendo Game Boy+Color
Prepare Python environs for OpenAI retro video game Reinforcement Learning integration
Experiment with SerenadeAI and prototype ChatGPT APIs across multiple language range

Note: leveraging Sega Master System development setup for Nintendo Game Boy is an achievement!

2024 Objectives

Resume Microservices application development and deploy across Distributed Systems at scale
Apply Infrastructure as Code and adopt DevOps best practices as software grows in complexity
Extend Software Engineering experience across Design + Architectural responsibilities in Cloud
Integrate Artificial Intelligence to Software projects: Machine, Deep or Reinforcement Learning

Architecture
Software architecture plays crucial role in designing + developing effective software systems. Understanding various architectural styles + patterns available, you can make informed decisions in architectural endeavors to create scalable, maintainable, and reliable software solutions. Architectural styles are high-level strategies that provide an abstract framework for a family of systems e.g. Layered, Event-Driven, and Microservices.

Architect Types
Software architects, solution architects and enterprise architects all play a part in designing, developing and implementing solutions that meet the needs of the business but differ with technical, company and IT focus:

Software Architect	A professional who designs and oversees the development of each software system. Ensures that the system meets the needs of the business and is technically feasible.
Solution Architect	A professional who designs and implements solutions that meet needs of the business. They have a deep understanding of both the business and the technology landscapes.
Enterprise Architect	A professional who designs and oversees the overall IT architecture of an organization. Ensures architecture aligned with business strategy and supports goals and objectives.

Cloud Variants
Cloud computing is becoming common for accessing data and using appliations in web-based environments. Growing demand for technology solutions in this area request need for cloud architects and cloud engineers:

Cloud Architect	A professional who deploys cloud-based applications according to business specifications. Architects focus work on the frameworks and infrastructure of cloud-based applications.
Cloud Engineer	A professional who designs, creates and develops cloud-based systems and solutions. Involved in building and testing technical requirements within all these cloud systems.

Microservices
Microservices have become a game-changer in the quickly changing technological landscape where flexibility, speed and scalability are crucial. Microservices architecture provides a more agile and effective substitute to conventional monolithic techniques and has seen rapidly increasing industry adoption in recent years.

Microservices align seamlessly with DevOps principles, fostering development and operations collaboration for continuous delivery and automation. Microservices architecture also aligns with containerization and orchestration tools like Kubernetes, simplifying the deployment and management of services at scale.

Industry predictions in Microservices expect growth to $1.7b by 2028 as 82% of organizations plan to adopt microservices in the next 3yrs. Microservices are well-suited to leverage emerging technologies such as IoT, edge computing, and blockchain and A.I. has potential to disrupt and innovate in the microservices space.

Future
The Top Tech Skills for 2024 corroborates this as specialization in the cloud computing market is predicted to grow at an annual rate of 18% plus Artificial Intelligence and Machine Learning continue to drive innovations across all industries including using AI in Microservices; AI is becoming an essential part of the microservices ecosystem, enabling businesses to build and maintain large, complex applications more efficiently. It will be interesting to see new innovative ways in which AI may integrate across the Software Engineering spectrum!

GBDK Programming Sample

noreply@blogger.com (SteveProXNA) — Wed, 15 Nov 2023 08:00:00 +0000

In the previous post, we checked out GBDK Programming Setup. The GBDK is a cross-platform development kit for sm83, z80 and 6502 based gaming consoles including the Nintendo Game Boy and Game Boy Color.

Using the GBDK, it is now possible to write game code using the C language rather than pure Z80 assembly. Therefore, we would like to extend this knowledge and checkout some code samples and homebrew games.

Let's check it out!

Software
Follow all instructions from the previous post: this documents how to setup all the pre-requisite software.
Note: ensure you have downloaded and installed the GBDK devkit and Small Device C Compiler [SDCC].

Examples
As per the previous post, after the GBDK has been downloaded, installed and setup, we should now checkout C:\gbdk\examples\gb to learn more about APIs available in the devkit. Replicate prior Hello World example.

Launch Visual Studio 2015. File | New | Project... | Visual C++ | Win32 | Win32 Project

Name:	Game
Location:	C:\apa_image
Create directory for solution	UNCHECKED

Copy over all original folders and files from the corresponding example. Copy from prior Hello World example the hidden .vscode folder and files. Copy also build.bat and align with source compile.bat file. Add relevant files to Visual Studio 2015. Press Ctrl+1 to build, link and run code similar to the prior Hello World example!

Launch Visual Studio Code. Open example folder. Set breakpoint in main.c. Press F5 to debug step thru code

Here is the complete list:

CUSTOM: apa_image	SOURCE: apa_image
CUSTOM: banks	SOURCE: banks
CUSTOM: bcd	SOURCE: bcd
CUSTOM: colorbar	SOURCE: colorbar
CUSTOM: comm	SOURCE: comm
CUSTOM: crash	SOURCE: crash
CUSTOM: dscan	SOURCE: dscan
CUSTOM: filltest	SOURCE: filltest
CUSTOM: galaxy	SOURCE: galaxy
CUSTOM: gb-dtmf	SOURCE: gb-dtmf
CUSTOM: gbcompress	SOURCE: gbcompress
CUSTOM: incbin	SOURCE: incbin
CUSTOM: irq	SOURCE: irq
CUSTOM: isr_vector	SOURCE: isr_vector
CUSTOM: large_map	SOURCE: large_map
CUSTOM: lcd_isr_wob	SOURCE: lcd_isr_wob

CUSTOM: linkerfile	SOURCE: linkerfile
CUSTOM: metasprites	SOURCE: metasprites
CUSTOM: paint	SOURCE: paint
CUSTOM: ram_function	SOURCE: ram_function
CUSTOM: rand	SOURCE: rand
CUSTOM: rpn	SOURCE: rpn
CUSTOM: scroller	SOURCE: scroller
CUSTOM: sgb_border	SOURCE: sgb_border
CUSTOM: sgb_multiplay	SOURCE: sgb_multiplay
CUSTOM: sgb_pong	SOURCE: sgb_pong
CUSTOM: sgb_sfx	SOURCE: sgb_sfx
CUSTOM: simple_physics	SOURCE: simple_physics
CUSTOM: sound	SOURCE: sound
CUSTOM: temp_minimal	SOURCE: temp_minimal
CUSTOM: temp_subfolder	SOURCE: temp_subfolder
CUSTOM: wav_sample	SOURCE: wav_sample

Name:	Game
Location:	C:\SimpleSprites
Create directory for solution	UNCHECKED

Copy over all original folders and files from the corresponding example. Copy from prior Hello World example the hidden .vscode folder and files. Copy also build.bat and align with source's make.bat file. Add relevant files to Visual Studio 2015. Press Ctrl+1 to build, link and run code similar to the prior Hello World example!

Launch Visual Studio Code. Open example folder. Set breakpoint in main.c. Press F5 to debug step thru code

Here is the complete list:

CUSTOM: helloworld	SOURCE: helloworld
CUSTOM: simplesprites	SOURCE: simplesprites
CUSTOM: movesprites	SOURCE: movesprites
CUSTOM: background	SOURCE: background
CUSTOM: window	SOURCE: window
CUSTOM: sound	SOURCE: sound
CUSTOM: noise	SOURCE: noise
CUSTOM: simplejumping	SOURCE: simplejumping
CUSTOM: metasprites	SOURCE: metasprites
CUSTOM: spritecollision	SOURCE: spritecollision
CUSTOM: bkgdcollision	SOURCE: bkgdcollision

CUSTOM: splashscreens	SOURCE: splashscreens
CUSTOM: fadeeffects	SOURCE: fadeeffects
CUSTOM: keyboard	SOURCE: keyboard
CUSTOM: savename	SOURCE: savename
CUSTOM: colorsprites	SOURCE: colorsprites
CUSTOM: backgrounds	SOURCE: backgrounds
CUSTOM: makingmusic	SOURCE: makingmusic
CUSTOM: spritetricks	SOURCE: spritetricks
CUSTOM: scrolling	SOURCE: scrolling
CUSTOM: debugging	SOURCE: debugging
CUSTOM: platformer	SOURCE: platformer

Homebrew
Larold's Jubilant Junkyard is a website/blog/shop which focuses on creating digital products including game development projects and tutorials. The Nintendo GameBoy development tutorial series is very educational.

Follow Larold's Jubilant Junkyard YouTube video channel for additional GBDK 2020 information and content:

SCREEN resolution	20 x 18 tiles	160 x 144 pixels
BACKGROUND	32 x 32 tiles	256 x 256 pixels

Compiler flags

-Wm-yc	GameBoy Color compatible	NB: lowercase c
-Wm-yC	GameBoy Color only	NB: uppercase C
-Wm-ys	Super GameBoy compatible

GameBoy Color Tile Attributes

Bit 7	priority flag	0: tiles below sprites, 1: tiles above sprites
Bit 6	vertical flip	0: tiles normal, 1: flipped vertically
Bit 5	horizontal flip	0: tiles normal, 1: flipped horizontally
Bit 4	not used
Bit 3	character bank specification	0: tiles at bank0, 1: tiles at bank1
Bit 2	see bit 0
Bit 1	see bit 0
Bit 0	bits 0-2	indicate tile color palette assignment

Name:	Game
Location:	C:\AlleywayGameboy
Create directory for solution	UNCHECKED

Copy over all original folders and files from the corresponding example. Copy from prior Hello World example the hidden .vscode folder and files. Copy also build.bat and align with source's make.bat file. Add relevant files to Visual Studio 2015. Press Ctrl+1 to build, link and run code similar to the prior Hello World example!

NOTE: ensure that the GBDK header files are included for Intellisense: Right click project | Properties |
Configuration Properties | C/C++ | General | Additional Include Directories | %GBDK_WIN%/include; $(SolutionDir)/headers/gen; $(SolutionDir)/headers/main; %(AdditionalIncludeDirectories).

Here is the complete list:

CUSTOM: alleyway	SOURCE: alleyway
CUSTOM: flappy-bird	SOURCE: flappy-bird

CUSTOM: pacman	SOURCE: pacman
CUSTOM: space-invaders	SOURCE: space-invaders

Summary
Armed with all this knowledge, we are now in an excellent position to build complete video games for the Nintendo Game Boy and Game Boy Color. In the interim, we see the GameBoy Competition 2023 is here!

GBDK Programming Setup

noreply@blogger.com (SteveProXNA) — Fri, 15 Sep 2023 09:00:00 +0000

In 2017, we checked out devkitSMS Programming Setup to build 8-bit video games for Sega Master System. Here we installed all the tools required including a powerful emulator: Emulicious to test the video games.

Emulicious is a free multi-system emulator popular for Sega Master System and Sega Game Gear games but can also play Nintendo Game Boy and Game Boy Color games. Therefore, let's leverage Sega Master System development setup to build and test games for Nintendo Game Boy and Game Boy Color using the GBDK.

Let's check it out!

GDDK
The GBDK is a cross-platform development kit for sm83, z80 and 6502 based gaming consoles including the Nintendo Game Boy and Game Boy Color. It includes libraries, toolchain utilities and SDCC C compiler suite.

Software
Follow all instructions from the previous post: this documents how to setup the pre-requisite software.

Here is a summary of all required software to be installed:

Name	Version
C IDE Editor	Visual Studio 2015
C IDE Editor	Visual Studio Code
Cross compiler	Small Device C Compiler

Name	Version
Make files	Cygwin
Emulators	Emulicious, BGB
Debugger	Emulicious Debugger

Note: this post documents how to download and install SDCC v4.1.0 on both Windows and Linux.

GBDK
Navigate to the gbdk-2020 repository on github: gbdk-2020 has full instructions here. Download the latest release from gbdk-2020 archive e.g. gbdk-4.1.1 or git clone latest source code. Extract code into C:\gbdk.

Setup the following environment variable for GBDK_WIN as it can be useful throughout GBDK development. Add the following one environment variable: System | Advanced system settings | Environment Variables:

Variable	Value	Description
GBDK_WIN	C:\gbdk	Windows path format

Example
As an example, let's write a simple program that prints Hello World to the screen using the GameBoy DevKit. Create new directory: C:\HelloWorld. Create new file main.c. Copy in the following code to print Hello World.

main.c

#include <gb/gb.h>
#include <stdio.h>

void main()
{
	printf( "HELLO WORLD" );
}

Build
Manually compile and link the Hello World program. Launch command prompt: Start | Run | cmd.

Change directory cd C:\HelloWorld. Next, execute the following 2x commands (in bold):

ACTION	COMMAND	OUTPUT
Compile	%GBDK_WIN%/bin/lcc -Wa-l -Wl-m -Wl-j -c -o main.o main.c	main.o
Link	%GBDK_WIN%/bin/lcc -Wa-l -Wl-m -Wl-j -o output.gb main.o	output.gb

Finally, type output.gb. The Hello World program should launch in the Emulicious emulator.
Congratulations! You have just written your first test program using the GameBoy DevKit.

IMPORTANT
If game does not initially render correct then select Options menu | System | Game Boy or Game Boy Color.

Automate
Let's automate the build process: create C:\HelloWorld\build.bat script file that contains the commands:

@echo off
%GBDK_WIN%/bin/lcc -Wa-l -Wl-m -Wl-j -c -o main.o main.c
%GBDK_WIN%/bin/lcc -Wa-l -Wl-m -Wl-j -o output.gb main.o
output.gb

Visual Studio 2015
Similar to Sega Master System development using devkitSMS, setup Visual Studio 2015 as per instructions here. Also, follow instructions here to connect Ctrl+1 hot key to automatically compile, link and run code.

Launch Visual Studio 2015. File | New | Project... | Visual C++ | Win32 | Win32 Project

Name:	HelloWorld
Location:	C:\
Create directory for solution	UNCHECKED

Application type:	Console application
Additional options:	Empty project CHECKED

Navigate to C:\HelloWorld. Add existing: main.c and build.bat at the same folder level as HelloWorld.sln.

Finally, ensure that the GBDK header files are included for Intellisense: Right click project | Properties |
Configuration Properties | C/C++ | General | Additional Include Directories | %GBDK_WIN%/include.

Press Ctrl+1 to build, link and run!

Visual Studio Code
Similar to Sega Master System development using devkitSMS, setup Visual Studio Code as per instructions here. Add hidden .vscode folder + include tasks.json and launch.json files beneath to build + debug code.

Launch Visual Studio Code. Open folder C:\HelloWorld. Add the .vscode folder the following files beneath:

c_cpp_properties.json

{
  "configurations": [
    {
      "name": "stevepro",
      "intelliSenseMode": "clang-x64",
      "includePath": [
        "${env:GBDK_WIN}/include"
      ],
      "cStandard": "c11",
      "cppStandard": "c++17",
      "browse": {
        "path": [
          "${workspaceFolder}"
        ],
        "limitSymbolsToIncludedHeaders": true,
        "databaseFilename": ""
      }
    }
  ],
  "version": 4
}

tasks.json

{
    "version": "2.0.0",
    "label": "build",
    "type": "shell",
    "windows": {
        "command": "./build.bat"
      },
    "presentation": {"echo": true, "reveal": "always", "focus": false, "panel": "shared", "showReuseMessage": false, "clear": true},
    "group": {"kind": "build", "isDefault": true},
}

launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "emulicious-debugger",
            "request": "launch",
            "name": "Launch in Emulicious",
            "program": "${workspaceFolder}/output.gb",
            "port": 58870,
            "stopOnEntry": false,
        }
    ]
}

Open main.c. Set breakpoint. Press Ctrl + Shift + B to build source code. Press F5 to debug step thru code:

In Emulicious, choose Tools menu | Debugger to be able to fully debug step thru the Z80 assembly code:

Summary
Now the GBDK has been downloaded, installed and setup, we should checkout C:\gbdk\examples\gb to learn more about the APIs available in the devkit + checkout some code samples and complete homebrew games. This will be the topic of the next post.

eBPF Programming Sample

noreply@blogger.com (SteveProXNA) — Thu, 31 Aug 2023 08:00:00 +0000

In the previous post, we checked out eBPF Programming Setup. Extended Berkeley Packet Filter [eBPF] is technology that runs sandboxed programs in privileged context like the OS kernel to safely and efficiently extend the capabilities of the kernel at runtime without requiring changes to the kernel source code. One issue is there are currently limited methods to debug BPF programs thus we'd like to explore test options.

Let's check it out!

Software
Follow all instructions from the previous post to install CLion and VS Code on Ubuntu Linux. Also install the following components for low level C code unit testing, mocking and integration: Unity, CMock, Ceedling.

Unity	Unit testing framework built for C projects with a focus on working with embedded toolchains
CMock	Mock and stub generator and runtime for unit testing C designed to work smoothly with Unity
Ceedling	Build system for C projects using Ruby Rake [make] build system to integrate Unity + CMock

Pre-requisites
As above, Ceedling requires Ruby installed otherwise sudo: gem: command not found errors may be thrown. Also, CMock requires Ruby bundle be installed otherwise Command 'bundle' not found errors may be thrown.

sudo apt install ruby

sudo apt install ruby-bundler

Unity
Unity is a unit testing framework built for C, with a focus on working with embedded toolchains, which we would like to try and leverage for BPF development. Follow instructions here to install and configure Unity.

git clone git@github.com:ThrowTheSwitch/Unity.git
cd Unity
cmake .
make
sudo make install
sudo cp libunity.a /usr/local/lib
sudo cp src/unity*.h /usr/local/include/unity

CMock
CMock is a mock and stub generator designed to work smoothly with Unity. CMock automagically parses the C headers and creates useful mock interfaces used for unit testing. Follow instructions here to install CMock.

git clone --recursive https://github.com/throwtheswitch/cmock.git
cd cmock
bundle install

Ceedling
Ceedling is a build system for C projects using Ruby Rake [make] build system. Ceedling also makes TDD Test Driven Development simple by integrating CMock + Unity. Follow instructions here to install Ceedling.

ruby -v
sudo gem install ceedling

Hello Unity
In the previous post, examples installed the libbpf C-based library. Therefore, extend an example and write tests that invoke simple libbpf APIs, for example, libbpf_num_possible_cpus directly using Unity framework.

Launch CLion | New Project | C Executable | Location: /home/stevepro/HelloUnity | Language std: C11

Create

Organize src and test directory structure. Install libbpf beneath src folder. Rename main.c to all_tests.c and move beneath test folder. Update all prod and unit test code. Complete CMakeLists.txt to include library files:

cmake_minimum_required(VERSION 3.19)
project(HelloUnity C)
set(CMAKE_C_STANDARD 11)

include_directories(src/ /usr/local/include/unity/ src/include/libbpf/src)
link_directories(/usr/local/lib/ src/include/libbpf/src/amd64)

file(GLOB CODE_FILES src/*.c)
file(GLOB TEST_FILES test/*.c)
add_executable(HelloUnity ${CODE_FILES} ${TEST_FILES} )
target_link_libraries(HelloUnity unity bpf elf z)

Edit configurations | Run with root privileges | Set breakpoints | Press F5 to debug step thru source code.

Hello CMock
At the time of writing, there is currently no way to debug BPF programs' source code interactively. Therefore, we would like to attempt to mock BPF programs or at least underlying C sourc code somehow using CMock.

Follow the example Spin up CMock. This article uses Ceedling to automate project setup and generate mocks but we will complete the following manually first to better understand the process. Original code found here.

Open folder /home/stevepro/HelloCMock. Copy inc/rectangle.h + project.yml. Execute mock command:

ruby ~/GitHub/ThrowTheSwitch/cmock/lib/cmock.rb -oproject.yml inc/rectangle.h

This creates mocks folder two corresponding files: mock_rectangle.h and mock_rectangle.c. Notice we did not attempt to use CLion as is not currently integrated; any attempt may produce Multiple Definition issue!

Hello Ceedling
Complete previous example but this time try Ceedling using the article as a guideline: Launch the Terminal:

cd $HOME
ceedling new HelloCeedling

Copy all project files into src and test directories. Include inc and update project.yml with the inc directory:

  :paths:
  :test:
    - +:test/**
    - -:test/support
  :source:
    - src/**
  :include:
    - inc/**
  :support:
    - test/support
  :libraries: []

Run tests manually simply by typing ceedling

Debug tests interactively with GDB as follows: Launch VS Code. Press F5 to add launch.json file to complete:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "ceedling_gdb",
            "type": "cppdbg",
            "request": "launch",
            "program": "${workspaceFolder}/build/test/out/test_shape_container.out",
            "args": [],
            "stopAtEntry": false,
            "cwd": "${workspaceFolder}",
            "environment": [],
            "externalConsole": true,
            "MIMode": "gdb",
            "miDebuggerPath": "/usr/bin/gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}

Set breakpoint in test_shape_container.c | Press F5 to debug step thru source code:

Summary
According to ChatGPT, eBPF is versatile technology that allows safe and efficient code execution within the Linux kernel without requiring modification of kernel code. This feature makes it attractive for applications including network packet filtering, security, logging + tracing, monitoring, and performance optimization.

In summary, while eBPF is a valuable addition to the Linux ecosystem, it is not expected to replace kernel development. Instead, eBPF complements this by currently providing a flexible and efficient way to extend specific kernel features. Therefore, it will be interesting to see if this technology is dominant in the future.

eBPF Programming Setup

noreply@blogger.com (SteveProXNA) — Tue, 04 Jul 2023 09:00:00 +0000

Extended Berkeley Packet Filter [eBPF] is a technology that runs sandboxed programs in privileged context like the OS kernel to safely and efficiently extend the capabilities of the kernel at runtime without requiring changes to the kernel source code. libbpf is a C-based library that takes compiled BPF object files and loads and verifies them into the kernel attachiing BPF programs to various kernel hooks and execute custom code.

Let's check it out!

Software
Follow all instructions from the previous post to install CLion and VS Code on Ubuntu Linux. Also install the bpftool which was added to the 2017 Linux kernel as a user space utility to introspect the eBPF subsystem.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
cd tools/bpf/bpftool
make
sudo make install
bpftool version

Alternatively, download bpftool binary from the Github releases and extract into /usr/local/bin directory.

Hello World
Create your first BPF program using this simple eBPF CO-RE application as a reference however using CLion. Launch CLion | New Project | C Executable | Location: /home/stevepro/HelloWorld | Language std: C11

Create

A new C project will be created which automatically adds the following two files: main.c and CMakeLists.txt

main.c

CMakeLists.txt

 #include <stdio.h>
 int main()
 {
   printf( "Hello, World!\n" );
   return 0;
 }

 cmake_minimum_required(VERSION 3.19)
 project(HelloWorld C)
 
 set(CMAKE_C_STANDARD 11)
 
 add_executable(HelloWorld main.c)

In the Terminal window within CLion IDE enter the following commands to install and buld the libbpf library:

git clone https://github.com/libbpf/libbpf
cd libbpf/src
make
sudo make install
cd ../..

Next, enter commands to create the vmlinux.h file. Create and build the Hello BPF program + skeleton file:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
touch hello.bpf.c
clang -g -O2 -target bpf -D__TARGET_ARCH_x86_64 -I libbpf/src -c hello.bpf.c -o hello.bpf.o
bpftool gen skeleton hello.bpf.o > hello.skel.h

hello.bpf.c

CMakeLists.txt

 #include "vmlinux.h"
 #include "bpf_helpers.h"
 
 SEC("tracepoint/syscalls/sys_enter_execve")
 int hello_bpf(struct trace_event_raw_sys_enter *ctx)
 {
   bpf_printk("Hello world!\n");
   return 0;
 }
 char LICENSE[] SEC("license") = "GPL";

 cmake_minimum_required(VERSION 3.19)
 project(HelloWorld C)
 
 set(CMAKE_C_STANDARD 11)
 
 include_directories(.)
 link_directories(libbpf libbpf/src)
 
 add_executable(HelloWorld main.c)
 target_link_libraries(HelloWorld bpf elf z)

Rename main.c to hello.c and update the driver program code. Edit configurations | Run with root privileges

Set break point in hello.c | HelloWorld | Debug or Press F5 to launch your first Hello World BPF program!

IMPORTANT you can tail BPF output in Terminal window: sudo cat /sys/kernel/debug/tracing/trace_pipe

Hello Maps
BPF maps are generic structures for storing different types of data accessible between kernel + user space. Launch CLion | New Project | C Executable | Location: /home/stevepro/HelloMaps | Language std: C11

Create

Repeat the process similar to Hello World: Install the libbpf library. Create the vmlinux header file, the Maps BPF program and driver code. Compile code. Update the CMakeLists build script and Run with root privileges:

libbpf-bootstrap
libbpf-bootstrap provides a starting point for building your own BPF applications with things like BPF Compile Once-Run Everywhere [CO-RE] and vmlinux.h. Numerous more BPF samples can be found at the Linux repo.

CUSTOM: minimal	SOURCE: minimal
CUSTOM: minimal_ns	SOURCE: minimal_ns
CUSTOM: minimal_legacy	SOURCE: minimal_legacy
CUSTOM: bootstrap	SOURCE: bootstrap
CUSTOM: uprobe	SOURCE: uprobe
CUSTOM: usdt	SOURCE: usdt

CUSTOM: fentry	SOURCE: fentry
CUSTOM: kprobe	SOURCE: kprobe
CUSTOM: ksyscall	SOURCE: ksyscall
CUSTOM: tc	SOURCE: tc
CUSTOM: profile	SOURCE: profile
CUSTOM: sockfilter	SOURCE: sockfilter

Summary
Although libbpf allows BPF programs to be loaded, verified and attached to various kernel hooks and execute custom code there is currently no way to debug BPF programs' source code interactively. Therefore, this now presents alternatives to explore unit testing and mock BPF programs. This will be the topic of the next post.

Sega Console Game Port

noreply@blogger.com (SteveProXNA) — Thu, 01 Jun 2023 07:07:00 +0000

In 2017, we checked out devkitSMS Programming Setup to build code in C for the 8-bit Sega Master System. Then, in 2020, we checked out SGDK Programming Setup to build code in C for the 16-bit Sega Mega Drive.

Leveraging this knowledge, we would now like to port existing code built for the 8-bit Sega Master System, for example, Simpsons Trivia SMS to the 16-bit Mega Drive Simpsons Trivia MD and document the process.

SEGA MASTER SYSTEM [256 x 192]	SEGA MEGA DRIVE [320 x 224]

Let's check it out!

Software
Follow most instructions from the previous post: this documents how to setup some pre-requisite software.

Here is a summary of some of the software to be installed:

Name	Version
C IDE Editor	Visual Studio 2015
Emulators	Fusion, Gens KMod

Name	Version
Cross compiler	GCC
Make files	Cygwin

Setup Project
Create folder C:\SimpsonsTriviaMD. Copy new SGDK custom library lib folder here. Create sub-folder: dev. Change directory to dev folder and create the following sub-folders as indicated per Usage: inc, out, res, src

Launch Visual Studio 2015. File | New | Project... | Visual C++ | Win32 | Win32 Project

Name:	Game
Location:	C:\SimpsonsTriviaMD\dev
Create directory for solution	UNCHECKED

Application type:	Console application
Additional options:	Empty project CHECKED

Finish

First, remove x64 build configuration! Right click Solution | Properties. Click Configuration Manager button. Click Active solution platform drop down | Edit. Click x64 option | Remove. Now will have only Win32 build.

Right click Project | Properties | Configuration Properties | General. Set Output and Intermediate directories:

Output Directory:	$(SolutionDir)..\bin\$(ConfigurationName)
Intermediate Directory:	$(SolutionDir)..\obj\$(ConfigurationName)

Right click Project | Properties | Configuration Properties | C++ | General. Set Additional Include Directories:

Value:

$(SolutionDir)..\lib;$(ProjectDir)inc;$(ProjectDir)res;$(GDK_WIN)\inc;$(IncludePath)

Value:

$(SolutionDir)..\lib

Right click Project | Properties | Configuration Properties | Linker | Input. Set the Additional Dependencies:

Value:

_genesis.lib;%(AdditionalDependencies)

The final solution should look like this:

Setup Resources
Import all resources into your project as media files that your game will use such as images for background tiles + sprites, audio for music + sound effects etc. Clone the "Setup Project" folder and follow these steps:

Change directory to res folder under dev folder. Copy all resources here. Create corresponding *.res files. Next execute build.bat to compile all resources then generate all corresponding gfx and sfx header files.

Create folder gfx at same level as dev folder. Create sub folder res. Add dummy corresponding translation unit files under res sub folder. Repeat process with sfx folder. Import all translation units into Visual Studio.

Setup Folders
Larger game projects will require nested folders in the inc or src directories to organize engine and screen code etc. Clone the "Setup Resources" folder and add all necessary subfolders + then update the makefile.

Launch Visual Studio and include resource files as before. Next, scale out engine and screen code but now organize into subfolders. Create as New Filters under Visual Studio Header Files and Source Files solution:

Create corresponding engine and screen subfolders beneath inc folder under dev. Add header files. Create engine and screen subfolders beneath src folder under dev. Add all the corresponding translation unit code.

Right click Project | Properties | Configuration Properties | C++ | General. Set Additional Include Directories:

Value:

$(SolutionDir)..\lib;$(ProjectDir)inc;$(ProjectDir)res;$(GDK_WIN)\inc;$(IncludePath);
$(ProjectDir)inc\engine;$(ProjectDir)inc\object;$(ProjectDir)inc\screen

Customize the makefile.gen file to include the new subfolders accordingly. Copy %GDK_WIN%\makefile.gen to local dev folder. Update makefile: Add new $(wildcard $(SRC)/ and -I$(INCLUDE) entries per folder.

Update dev built.bat file to use new local custom makefile.gen instead of de-facto %GDK_WIN% version.
If you remove pre-build step to speed up makefile build then don't forget to create out subfolders manually.

Convert APIs
Finally, copy over all engine, object, screen etc. files from the Sega Master System code base into the Sega Mega Drive port. Convert all devkitSMS APIs to the SGDK equivalents. Allow for different screen resolution!

Here is quick cheat of devkitSMS to SGDK API conversions; of course not all APIs are one-to-one mapping:

	SEGA MASTER SYSTEM	SEGA MEGA DRIVE
Audio	PSGPlay(); PSGStop(); PSGGetStatus();	XGM_startPlayPCM(); XGM_stopPlayPCM(); XGM_isPlayingPCM();
Dimensions	256 x 192 [32x24]	320 x 224 [40x28]
Font	SMS_setNextTileatXY(); SMS_setTile();	VDP_setMapEx();
Graphics	SMS_loadPSGaidencompressedTiles(); SMS_loadBGPalette();	VDP_drawImageEx(); VDP_setPalette();
Hack	Address : 0x0050	Address : 0x01E0
Input	SMS_getKeysStatus();	JOY_readJoypad();
Random	rand();	random();
Sprites	SMS_addSprite(); SMS_copySpritestoSAT();	SPR_addSprite(); SPR_update();

Summary
Repeat process with Skazka. In summary, the Master System to Mega Drive port should be fairly seemless J

SEGA MASTER SYSTEM [256 x 192]	SEGA MEGA DRIVE [320 x 224]