anti patterns codin'

Fixing or Repairing Windows Installation and Checking Disks for Errors

2025-05-14T15:56:00.002+01:00

To fix or repair Windows (for example if problems with Microsoft Teams):

sfc /scannow

DISM.exe /Online /Cleanup-image /Restorehealth

To check disks for bad clusters on C drive, and try to fix:

chkdsk /x/r c:

tips:

- you should always back up your data, just in case.

- all media are subject to 'bit rot' over time, including optical media.

- consider holding data on a separate partition from the operating system.

- consider backing up to an offsite target.

- consider backing up the 'raw' original files, so that they are easy to restore.

REST API guidelines from Microsoft

2025-03-05T08:38:00.004+00:00

REST API guidelines from Microsoft

This article notes some REST API guidelines from Microsoft (they are applicable to other tech stacks too, such as Python FastAPI). The Microsoft article has further guidelines, this article presents a selection.

Organize the API design around resources

- when designing the URL paths, think in terms of 'objects' rather than verbs

Examples:

https://adventure-works.com/orders // Good

https://adventure-works.com/create-order // Avoid

ref: Web API design best practices - Azure Architecture Center | Microsoft Learn

Define API operations in terms of HTTP methods

- use GET and POST appropriately

- combine with URL 'paths' to model the hierarchy of objects

Examples:

/customers -> GET = Get all customers, POST = Create a new customer

/customers/1 -> GET = Get customer 1

/customers/1/orders -> GET = Get all orders for customer 1, POST = Create a new order for customer 1

Source

- Web API design best practices - Azure Architecture Center | Microsoft Learn

Pydantic: Dynamically creating a model, with field types, examples and descriptions.

2025-02-12T09:00:00.002+00:00

Pydantic: Dynamically creating a model, with field types, examples and descriptions.

Normally, Pydantic models are hard-coded and cannot change at runtime. However it is also possibly to build a Pydantic model 'on the fly'.

Here is a function to dynamically create a Pydantic model. Each field has a type, description and some examples:

from typing import Any, Tuple
from pydantic import BaseModel, Field, create_model

FieldDetails = Tuple[type, str, list[Any]]  # field type, description, examples

def create_dynamic_model(model_name:str, fields_dict: dict[str, FieldDetails]) -> type[BaseModel]:
    """
    Create a dynamic Pydantic model based on the provided fields dictionary.
    - why: we want to read the examples from example_store, not hard-code them.
    
    :param fields_dict: A dictionary where keys are field names and values are tuples of (type, description, examples)
    :return: A dynamically created Pydantic model
    """
    return create_model(
        model_name,
        **{
            field_name: (field_type, Field(..., description=field_desc, examples=examples))
            for field_name, (field_type, field_desc, examples) in fields_dict.items()
        }
    )

Example usage:

if __name__ == "__main__":
    # Define the fields for our dynamic model
    fields = {
        "name": (str, "The name of the person", ["Bob"]),
        "age": (int, "The age of the person", 54),
        "email": (str, "The email address of the person", "bob@hotmail.com"),
        "is_student": (bool, "Whether the person is a student or not", False),
    }

    # Create the dynamic model
    DynamicPerson = create_dynamic_model("Person", fields)

    # Create an instance of the model
    person = DynamicPerson(
        name="John Doe",
        age=30,
        email="john.doe@example.com",
        is_student=False,
    )

    # Print the model's schema
    print(DynamicPerson.schema_json(indent=2))

    # Print the instance
    print(person)

    # Access fields and their descriptions
    for field_name, field in DynamicPerson.model_fields.items():
        print(f"Field: {field_name}, Description: {field.description}")

Python: Convert a base64 encoded PDF file to a list of PIL (pillow) Images

2025-02-10T17:53:00.002+00:00

Python: Convert a base64 encoded PDF file to a list of PIL (pillow) Images

motivation: generally images are easier to work with than PDF files.

import base64
import pdf2image
from PIL import Image

def convert_base64_encoded_pdf_to_PIL_image(pdf_base64: str) -> list[Image.Image]:
    pdf_bytes = base64.b64decode(pdf_base64)
    return pdf2image.convert_from_bytes(pdf_bytes)

Problem: poetry does not pick up the current 'local' python (set via pyenv)

2025-01-29T10:01:00.006+00:00

Problem: poetry does not pick up the current 'local' python (set via pyenv)

pyenv is a nice Python version management tool.

Problem: it can happen that poetry does not pick up the current local Python version.

For example, if the global version is 3.11.7, then pyenv can be used to set the local version:

pyenv local 3.12.7

Then, using poetry (here it is used to run ruff, to format):

poetry run ruff format

There is an error, as the poetry environment is still using the global Python version:

Current Python version (3.11.7) is not allowed by the project (^3.12).

Please change python executable via the "env use" command.

Solution:

poetry env use python

This creates a new poetry environment, which picks up the local Python version.

The new environment needs its dependencies installed:

poetry install

Now, poetry should run OK:

poetry run ruff format

A starter template for a Python Command Line program

2025-01-24T16:03:00.003+00:00

A starter template for a Python Command Line program

There are several ways to create a command line program in Python.

This blog post presents a simple example 'template' based around the built-in optparse module (with OptionParser class). The optparse module has good documentation on how to set it up, but seems to lack a few examples of how to actually use it.

You can copy-paste this code and edit it to suit your particular command line program.

"""
main.py

TODO_DESCRIBE_WHAT_THIS_DOES.
"""

import sys
from optparse import OptionParser

# usage() - prints out the usage text, from the top of this file :-) and the options
def usage(parser):
    print(__doc__)
    parser.print_help()


# optparse - parse the args
parser = OptionParser(
    usage="%prog <TODO_ARGS> [options]",
    version="0.1.0"
)

parser.add_option(
    "-i", "--input", metavar="INPUT_FILE", help="read data from INPUT_FILE", dest="input_file"
)
parser.add_option(
    "-l",
    "--language",
    dest="target_language",
    metavar="TARGET_LANGUAGE",  # 'metavar' helps make 'help' messages refer to values and is used to build the help for this option.
    default="English",
    help="translate to the target output language TARGET_LANGUAGE. Example: English.",
)
parser.add_option(
    "-o",
    "--output",
    dest="output_dir",
    default=None,
    help="the output directory. By default is None, so output is to stdout (no files are output).",
)
parser.add_option(
    "-v", "--verbose",  action='store_true', help="turn on verbose mode", dest="is_verbose"
)

(options, args) = parser.parse_args()
if len(args) != 1:
    usage(parser)
    sys.exit(2)

first_arg = args[0]
target_language = options.target_language
path_to_output_dir = None
if options.output_dir:
    path_to_output_dir = options.output_dir

Running python main.py will output help text like:

main.py

TODO_DESCRIBE_WHAT_THIS_DOES.

Usage: main.py <TODO_ARGS> [options]

Options:

--version show program's version number and exit

-h, --help show this help message and exit

-i INPUT_FILE, --input=INPUT_FILE

read data from INPUT_FILE

-l TARGET_LANGUAGE, --language=TARGET_LANGUAGE

translate to the target output language

TARGET_LANGUAGE. Example: English.

-o OUTPUT_DIR, --output=OUTPUT_DIR

the output directory. By default is None, so output is

to stdout (no files are output).

-v, --verbose turn on verbose mode

How to read and write a Pydantic class to/from a JSON file.

2025-01-22T10:21:00.002+00:00

How to read and write a Pydantic class to/from a JSON file.

Pydantic classes are (de)serializable, although deserializing can be a little tricky.

Here is some example code, with functions to load and save data for an example Pydantic class 'MySerializableType':

import logging
from cornsnake import util_file, util_json
from pydantic import BaseModel, TypeAdapter

logger = logging.getLogger(__file__)

class CustomBaseModel(BaseModel):
    class Config:
        populate_by_name = True
        extra = "forbid"

class MySerializableType(CustomBaseModel):
    favourite_color: str = "blue"
    file_schema_version: str = "0.2"

def read_from_file(path_to_file: str) -> MySerializableType:
    try:
        json_data = util_json.read_from_json_file(path_to_file)
        deserialized =  TypeAdapter(MySerializableType).validate_python(json_data)
        return deserialized
    except Exception as e:
        logger.exception(e)


def save_to_file(path_to_file: str, data: MySerializableType) -> None:
    json_data = data.model_dump_json(by_alias=True)
    util_file.write_text_to_file(json_data, path_to_file)

notes:

- read_from_file(): TypeAdapter is used to validate and map the JSON back to the expected Pydantic class

- save_to_file(): model_dump_json() is used to serialize the Pydantic object to JSON. The parameter by_alias=True ensures that the BaseModel config options are followed (in particular, if camelCasing is specified as opposed to snake_case).

Python Debugging Tips

2024-12-14T10:00:00.002+00:00

Coding a breakpoint, which activates the REPL debugger:

import pdb
pdb.set_trace()

Printing out the stack trace of an exception:

import traceback
try:
print(variable_that_does_not_exist)
except Exception as e:
print(traceback.format_exc())

import traceback
try:
    print(variable_that_does_not_exist)
except Exception as e:
    print(traceback.format_exc())

import traceback
try:
    print(variable_that_does_not_exist)
except Exception as e:
    print(traceback.format_exc())

import traceback
try:
    print(variable_that_does_not_exist)
except Exception as e:
    print(traceback.format_exc())

Adding custom 'summary' charts to MLflow: simple bar chart and 'grouped' bar chart

2024-12-01T11:10:00.005+00:00

MLflow is a convenient and OSS (Open Source Software) service for accepting the result of a machine learning evaluation.

By default, the MLflow user interface can be used to add basic charts of metrics, to view results and also compare across runs.

However if there are many metrics, then it can be useful to create a custom 'summary' chart.

The approach taken here is to use matplotlib to generate custom charts, and then attach as an artifact using the MLflow API. This follows the MLflow documentation.

note: To allow for grouping of metrics by category, the following code assumes a convention where the MLflow metrics are named like '<category>__<metric>'.

So the data being logged to MLflow looks like this:

[('category1__response_time_ms', 11000.0), ('category2__response_time_ms', 20000.0)]

Simple Bar Chart

The following code shows how to generate a summary bar chart, with one bar for each metric:

import logging
import typing
from matplotlib.colors import Colormap
from matplotlib.figure import Figure
import pandas as pd
from matplotlib import pyplot as plt
import numpy as np

logger = logging.getLogger(__name__)


MLflowFigure = tuple[str, Figure]  # title and figure
MetricAndMeasure = tuple[str, float | None]

METRIC_NAME_DIVIDER = "__"  # MLflow attributes have been setup to have a category, by using a prefix like: metric_name = <category>__<metric>


def _build_color_map_by_value(values: list[float]) -> Colormap:
    rescale = lambda y: (y - np.min(y)) / (np.max(y) - np.min(y))
    my_cmap = plt.get_cmap("viridis")

    return typing.cast(Colormap, my_cmap(rescale(np.array(list(values)))))


def _create_bar_chart(
    title: str, yaxis_title: str, measurements_by_category: dict[str, float], unit_short: str, is_debug: bool
) -> MLflowFigure:
    if is_debug:
        print(f"measurements_by_category = {measurements_by_category}")

    categories = [*measurements_by_category.keys()]
    values = [*measurements_by_category.values()]

    fig, ax = plt.subplots(layout="constrained", figsize=(21, 7))

    bar_width = 0.1
    ax.bar(
        x=categories,
        height=values,
        width=bar_width,
        align="edge",
        label=categories,
        color=_build_color_map_by_value(values=values),
    )

    x = np.arange(len(categories))  # the label locations
    bar_width = 0.1
    ax.set_xticks(x + bar_width, categories)

    # Add some text for labels, title and custom x-axis tick labels, etc.
    ax.set_ylabel(yaxis_title)
    ax.set_title(title)
    ax.legend(title=yaxis_title, ncols=len(categories))

    # label each bar with its value
    rects = ax.patches
    for i, rect in enumerate(rects):
        label = f"{values[i]}{unit_short}"
        height = rect.get_height()
        ax.text(rect.get_x() + rect.get_width() / 2, height, label, ha="center", va="bottom")

    # scale the y-axis
    y_max = 0.0
    for category in measurements_by_category:
        measurement = measurements_by_category[category]
        y_max = max(y_max, measurement)

    ax.set_ylim(0, y_max * 1.2)

    if is_debug:
        plt.show()

    return (title, fig)


def _create_summary_figure(mlflow_metrics: list[MetricAndMeasure], is_debug: bool) -> MLflowFigure:
    # example input: [('category1__response_time_ms', 11000.0), ('category2__response_time_ms', 20000.0)]
    metrics_by_category: dict[str, float] = {}
    metrics = {}
    for [category_and_metric, measurement] in mlflow_metrics:
        [category, metric] = category_and_metric.split(METRIC_NAME_DIVIDER)
        metrics[metric] = True
        metrics_by_category[category] = _from_millis_to_seconds(measurement)

    # assumption: there is just 1 time metric:
    if len(metrics) > 1:
        raise RuntimeError(f"Expected 1 time metric, but have: {metrics}")

    return _create_bar_chart(
        title="Response time by Prompt Category",
        yaxis_title="Response time (seconds)",
        measurements_by_category=metrics_by_category,
        unit_short="s",
        is_debug=is_debug,
    )


def create_figures(
    mlflow_metrics: list[MetricAndMeasure], is_debug: bool
) -> list[MLflowFigure]:
    return [
        _create_summary_figure(mlflow_metrics=mlflow_metrics, is_debug=is_debug),
    ]

This produces a bar chart like this, showing all the metrics for that run, side-by-side.

- the bars are colored according to their value (higher value = lighter hues)

Grouped Bar Chart

A more advanced technique is to take the categorized metrics (where the metric name is prefixed by a category) and then plot a grouped bar chart, with one group for each category. This allows for comparing categories across multiple metrics (or attributes).

import logging
from matplotlib.figure import Figure
from matplotlib import pyplot as plt
import numpy as np

logger = logging.getLogger(__name__)


MLflowFigure = tuple[str, Figure]  # title and figure
MetricAndMeasure = tuple[str, float | None]


def _create_grouped_bar_chart(
    title: str,
    yaxis_title: str,
    categories: list[str],
    grouped_metrics: dict[str, list[float]],
    is_debug: bool,
    decimal_places: int = 2,
) -> MLflowFigure:
    x = np.arange(len(categories))  # the label locations
    bar_width = 0.1
    multiplier = 0
    gap_between_bars = 0.02

    fig, ax = plt.subplots(layout="constrained", figsize=(21, 7))

    for attribute, measurement in grouped_metrics.items():
        offset = (bar_width + gap_between_bars) * multiplier
        measurement = [round(y, decimal_places) for y in measurement]

        rects = ax.bar(x + offset, measurement, bar_width, label=attribute)
        ax.bar_label(rects, padding=3)
        multiplier += 1

    # Add some text for labels, title and custom x-axis tick labels, etc.
    ax.set_ylabel(yaxis_title)
    ax.set_title(title)
    ax.set_xticks(x + bar_width, categories)
    ax.legend(loc="upper left", ncols=len(grouped_metrics.keys()))

    y_max = 0.0
    for metric in grouped_metrics:
        for value in grouped_metrics[metric]:
            y_max = max(y_max, value)

    ax.set_ylim(0, y_max * 1.2)

    if is_debug:
        plt.show()

    return (title, fig)


METRIC_NAME_DIVIDER = "__"


def _create_accuracy_summary_figure(accuracy_metrics: list[MetricAndMeasure], is_debug: bool) -> MLflowFigure:
    # example input: [('category1__entity_count_accuracy', 1.0), ('category1__attribute_count_accuracy', 1.0)]

    grouped_and_named_metrics: dict[str, dict[str, float | None]] = {}

    all_metrics: list[str] = []
    for category_and_metric in accuracy_metrics:
        if METRIC_NAME_DIVIDER in category_and_metric[0]:  # like 'category1__attribute_count_accuracy'
            [category, metric] = category_and_metric[0].split("__")
            if category not in grouped_and_named_metrics:
                grouped_and_named_metrics[category] = {metric: category_and_metric[1]}
            else:
                existing = grouped_and_named_metrics[category]
                existing[metric] = category_and_metric[1]
            all_metrics.append(metric)
        else:  # like 'overall_accuracy'
            category = category_and_metric[0]
            metric = category_and_metric[0]
            grouped_and_named_metrics[category] = {metric: category_and_metric[1]}
            all_metrics.append(metric)

    # To have same shape of data for plotting, fill in any missing metrics as zero:
    for metric in all_metrics:
        for category in grouped_and_named_metrics:
            existing = grouped_and_named_metrics[category]
            if metric not in existing:
                existing[metric] = 0.0

    # project to the structure required by plot:
    shared_metrics: list[str] = []
    grouped_metrics: dict[str, list[float]] = {}

    if is_debug:
        print(f"grouped_and_named_metrics = {grouped_and_named_metrics}")

    categories = []

    for category in grouped_and_named_metrics:
        categories.append(category)
        sorted_metrics = list(grouped_and_named_metrics[category].keys())
        sorted_metrics.sort()  # sort the metrics to ensure they are consistent across categories
        if not shared_metrics:
            shared_metrics = sorted_metrics
        elif shared_metrics != sorted_metrics:
            raise RuntimeError(
                f"Figure data is misaligned: shared_metrics={shared_metrics}, new metrics={sorted_metrics}"
            )
        for metric in sorted_metrics:
            if metric not in grouped_metrics:
                grouped_metrics[metric] = []
            value = grouped_and_named_metrics[category][metric]
            if not value:
                value = 0.0
            grouped_metrics[metric].append(value)

    if is_debug:
        print(f"categories: {categories}")
        print(f"metrics: {shared_metrics}")
        print(f"grouped_metrics: {grouped_metrics}")

    return _create_grouped_bar_chart(
        title="Accuracy by Prompt Category",
        yaxis_title="Accuracy",
        categories=categories,
        grouped_metrics=grouped_metrics,
        is_debug=is_debug,
    )


def create_figures(
    mlflow_metrics: list[MetricAndMeasure], is_debug: bool
) -> list[MLflowFigure]:
    return [
        _create_accuracy_summary_figure(accuracy_metrics=mlflow_metrics, is_debug=is_debug),
    ]

This produces a grouped bar chart, similar to this:

note: since the code involves some projection of data, it can be useful to cover this with unit tests.

Summary

Creating custom figures for MLflow is a powerful technique to gain more insights into your ML evaluation results, and better communicate the results to colleagues.

For more information, please see the MLflow documentation.

Show the source code of a Python module (also in Jupyter)

2024-10-11T18:02:00.001+01:00

How to show the source code of Python module (also in Jupyter)

- using the inspect module

Example: to show the source code for a 'utils' module:

import utils
import inspect

lines = inspect.getsource(utils)
print(lines)

# in Jupyter, use display:
# display(lines)

Articles about tests involving 3rd party services - End-to-end instead of Integration or Contract tests

2024-09-05T16:13:00.003+01:00

Integration tests for 3rd part APIs

https://axelhodler.medium.com/integration-tests-for-third-party-apis-dab67c52e352

Regression and Functional tests are meaningless

https://gerg.dev/2020/08/regression-and-functional-tests-are-meaningless/

Testing 3rd party services

https://www.reddit.com/r/ExperiencedDevs/comments/1aoh9v4/testing_3rd_party_services/

Exporting a Jupyter notebook without the input code

2024-06-26T14:29:00.002+01:00

1. export to HTML

2. open in a browser (Edge or Chrome)

3. paste JavaScript code into the console, to remove the input fields:

document.querySelectorAll("div.jp-InputArea.jp-Cell-inputArea").forEach(function(a){a.remove()})

- see https://stackoverflow.com/questions/34818723/export-notebook-to-pdf-without-code

4. save as HTML, or print to PDF

World maps in Python - free map resources

2024-06-21T15:33:00.004+01:00

Map files (ZIP):

Natural Earth - Free vector and raster map data at 1:10m, 1:50m, and 1:110m scales (naturalearthdata.com)

The code below uses "Admin 0 – Countries" although other map files should also work, if they have countries in their 'NAME' column.

Python:

Installing packages [geopandas] [seaborn]:

pip install geopandas seaborn

Loading the world map:

import geopandas as gpd
# load the low resolution world map
world = gpd.read_file("./resources/ne_110m_admin_0_countries.zip") # ref https://www.naturalearthdata.com/downloads/110m-cultural-vectors/
all_countries = world['NAME']

def prepare_world_map_countries(country_names_to_highlight):
countries_joined = '|'.join(country_names_to_highlight))
countries_joined

countries = world[world['NAME'].str.contains(countries_joined) == True]
countries = list(countries['NAME'])
return countries

Plotting the world map:

import matplotlib.patches as mpatches
import seaborn as sns

def plot_world_map(countries, title):
# initialize a new figure
fig = plt.figure(figsize=(20, 10))
ax = fig.add_subplot()

colors = sns.color_palette("husl", len(all_countries))
patches = []

# loop over the continent names and corresponding colors
for (country_name, color) in list(zip(all_countries, colors)):
# grab all countries that belong to the continent, then plot each of the
# continents, giving each a unique color
continent = world[world["NAME"] == country_name]
is_active_country = country_name in countries
actual_color = color if is_active_country else 'white'
continent.plot(ax=ax, color=actual_color, edgecolor="black", alpha=0.5)

# generate a patch for the current continent
patch = mpatches.Patch(label=country_name, color=color)
if is_active_country:
patches.append(patch)

# add the patches to the map
ax.legend(handles=patches, loc="lower left", ncol=4)

# turn off axis ticks
ax.set_xticks([])
ax.set_yticks([])

# set the plot title
plt.title(title)
plt.show()

Example execution:

country_names_to_highlight = ['United States of America',
'South Africa',
'Brazil',

'China',

'Australia',

'Colombia',
'France']
plot_world_map(country_names_to_highlight, "Countries Visited")

Resulting map:

Avoiding SettingsWithCopyWarning from pandas when adding a column as function of other columns

2024-06-10T11:11:00.005+01:00

Adding a new column as a function of other columns can hit performance issues, and more seriously can be unreliable if implemented incorrectly.

For simple cases, a vector based filter can efficiently add the values of the new column, via loc():

df.loc[df['eri_white']==1,'race_label'] = 'White'

However more complicated cases can require a function that checks each cell value.

Example:

def get_label(text):

if is_css(text):

return 'css'

if is_json(text):

return 'json'

return 'other'

Applying this function via 'apply' and then using that result to assign to a new column, triggers the SettingWithCopyWarning warning from pandas:

dfSourceLanguages['label'] = dfSourceLanguages.apply(lambda r: get_label(r['text']), axis=1)

The SettingWithCopyWarning of pandas catches this and other cases where the code is not efficient and might not be reliable:

SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy dfSourceLanguages['label'] = dfSourceLanguages.apply(lambda r: get_label(r['text']), axis=1)

This warning also occurs when using map():

dfSourceLanguages['label'] = dfSourceLanguages['text'].map(get_label)

---

Another approach: use df.assign()

# Assigns a new column 'label' via the lambda function

# - applies _get_label() to the series 'row.text'

df = df.assign(

label = lambda row: row.text.apply(_get_label)

)

Expanding columns and rows in pandas results in Jupyter notebook

2024-06-10T08:51:00.004+01:00

Expanding column widths

By default, when executed in a Jupyter notebook, pandas results have their columns truncated to about 200px.

This sometimes obscures interesting parts of the results

One way to expand the column widths is to increase the max-width:

from pandas import option_context
with option_context('display.max_colwidth', 400):
    display(df.head())

This example expands columns that have long values, up to 400px.

Alternatively, select a small number of rows (for example via head()) and print the transpose:

display(df.head().T)

Expanding row counts

similarly, more rows can be displayed via this setting:

pd.set_option('display.max_rows', 500) 

Converting a Python array to a pandas dataframe [multiplying data]

2024-06-09T11:51:00.009+01:00

Sometimes a Python array is simpler to generate or manipulate - but then we need to convert it to a Pandas dataframe for further processing:

import numpy as np

import pandas as pd

def convert_array_of_rows_to_dataframe(array_of_rows, columns):
  np_array = np.array(array_of_rows)
  dfOut = pd.DataFrame.from_records(np_array)
  dfOut.columns = columns
  return dfOut

Example call:

my_array = [ ['This is great', 'positive'],

['This is terrible', 'negative']

df = convert_array_of_rows_to_dataframe(my_array, ['text', 'label'])

A more complicated example is using Python code to 'multiply' data from CSV, and then convert that to a dataframe:


dfIn = pd.read_csv("../data/nouns.csv")
multiplied_rows = []
dfIn.reset_index()

phrases = [
"You are a NOUN", "What a NOUN", "Don't be such a NOUN",
"This waiter is a NOUN.",
"She was a NOUN today.",
"The company here are NOUNs.",
"That director was a NOUN."
]

for _, row in dfIn.iterrows():
  for template in phrases:
    multiplied_rows.append([template.replace("NOUN", row['text'].lower())])

dfMultiplied = convert_array_of_rows_to_dataframe(offensive_multiplied_rows, ['text'])


Here are some other articles about Pandas:
- https://www.kdnuggets.com/10-pandas-one-liners-for-exploratory-data-analysis

Ubuntu apt (and apt-get) tips

2024-05-22T12:59:00.004+01:00

Possibly apt is generally more suitable as a higher-level tool than apt-get:

The apt command line tool provides a higher-level user interface for end users with intuitive commands, resulting behaviors, and security features. In contrast, the command apt-get is a low-level interface that communicates more closely with core Linux processes.

apt package installer has a 'fix broken install' option that could be worth trying:

sudo apt --fix-broken install

Comparing Costs of Cloud Providers - vantage.sh

2024-04-14T09:24:00.008+01:00

vantage.sh is a free site that helps compare the costs of similar services on the main cloud providers.

Comparing Amazon AWS instances: https://instances.vantage.sh/

AWS RDS instances - https://instances.vantage.sh/rds/

AWS ElastiCache instances - https://instances.vantage.sh/cache/

AWS Redshift instances - https://instances.vantage.sh/redshift/

AWS OpenSearch instances - https://instances.vantage.sh/opensearch/

vantage.sh also provides a cost monitoring service, where you can integrate vantage.sh with your cloud environment - this is possible with many cloud providers (AWS, Azure...) - https://www.vantage.sh/

- note: the author has no affiliation with vantage.sh

Structured Logging (Semantic Logging) in Python and C#

2024-04-14T09:19:00.004+01:00

Why Structured logging:

1. machine parseable logs:

When a log is automatically parseable, this enables viewing, searching and filtering.

A parseable log can also enable further processing or display (for example: command line tools that output parseable logs are easier to automate via another tool such as a desktop app with graphical user interface).

2. logged events have context

Adding more context to each log event helps later debugging. Why: when programming, it can be difficult to know what info to add that would help future debugging.

Also see: the 'why' of Python's structlog.

Structured logging in C#:

serilog

serilog on github

Structured logging in Python:

structlog - Structured Logging for Python

Scanning Python code for security issues via Snyk: the ‘industrial strength’ option

2024-02-21T21:14:00.008+00:00

Scanning Python code for security issues via Snyk: the ‘industrial strength’ option

note: I am not at all affiliated to Snyk - just a fan of its quality features.

Snyk is a code scanning tool capable of detecting OWASP issues (Open Worldwide Application Security Project), following the CWE-NN standard (Common Weakness Enumeration) for identifying code security issues.

At time of writing, there is a free plan for individuals and small teams: https://snyk.io/plans/

Snyk supports many languages - here is the section for Python:

https://snyk.io/platform/snyk-python-security/

To add a Python repository to Snyk for scanning:

The project must have a setup.py and setup.cfg file, describing the project and its dependencies.

Note: normally setup.py and setup.cfg files are a modern way to describe how your code would be released as a Python package (for example to publish on pypi.org). However, even if you do not intend to publish your code as a package, I found it necessary to add these files.

Otherwise, I found Snyk does not recognise the project as a Python project.

A minimal setup.cfg file:

[egg_info] egg_base = . [tool:pytest] testpaths = .

note:

the [egg_info] section is used to specify the location of ‘egg package’ details on build
the [tool:pytest] section specifies where to look for unit tests

A minimal setup.py file, including the dependencies of the project:

import io from setuptools import find_packages from setuptools import setup with io.open("README.md", "rt", encoding="utf8") as f: readme = f.read() setup( name="my-app", description="My wonderful Python project.", long_description=readme, packages=find_packages(), include_package_data=True, zip_safe=False, install_requires=[], extras_require={"dev": ["flake8==6.1.0", "pyvan==1.2.2"]} )

Further reading about setup.py and setup.cfg files:

https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/

Good luck with your scanning!

Alternatives to Snyk

VeraCode: https://docs.veracode.com/r/Python_Scan_Directives?gad_source=1&gclid=CjwKCAiA29auBhBxEiwAnKcSqicM0d3L55z3LpysWl714-zWC_6ymqCxIXdrZDE1Q9kjxCMB5yv-TxoCkBIQAvD_BwE

SonarQube: https://docs.sonarsource.com/sonarqube/latest/analyzing-source-code/languages/python/

AWS Translate via the AWS CLI command line - error: 'charmap' codec can't encode characters

2024-02-14T16:25:00.004+00:00

With the AWS CLI tool, it is possible to perform machine translation on the command line:

aws translate translate-text --region "eu-west-1" --source-language-code "en" --target-language-code "es" --text "hello, world"

OUTPUT:

{
"TranslatedText": "hola, mundo",
"SourceLanguageCode": "en",
"TargetLanguageCode": "es"
}

notes:

1. Multiple AWS accounts via '--profile' option

- if you have multiple AWS accounts, you can add multiple credentials in your .aws/config file:

[default]
region = eu-west-1

[profile my-aws-1]
aws_access_key_id = xxx-1
aws_secret_access_key = yyy-1

[profile my-aws-2]
aws_access_key_id = xxx-2
aws_secret_access_key = yyy-2

- then, to use a particular AWS account, add the --profile option to AWS CLI tool:

aws translate translate-text --profile my-aws-1 --region "eu-west-1" --source-language-code "en" --target-language-code "es" --text "hello, world"

2. If you receive a character set error like this (especially if translating to a non-latin charachter set language like Spanish or Russian)

{ "TranslatedText": 'charmap' codec can't encode characters in position 1-6: character maps to <undefined>

Then, you need to set some environment variables to tell Python to use Unicode (the AWS CLI tool is written in Python):

On Windows:

set PYTHONIOENCODING=UTF-8

set PYTHONUTF8=1

3. It's also worth upgrading AWS CLI to the most recent version.

The installers are at: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Generate PDF from markdown via pandoc

2024-01-27T09:47:00.005+00:00

One option to generate PDF from a markdown file like README.md, is to use pandoc.

Installation

You need to install both:

- [Pandoc](https://pandoc.org/installing.html)

- [MiKTeX](https://miktex.org/download)

Then, restart any terminal windows where you will run the command to generate PDF.

Execution - basic command

pandoc -o ./README.pdf ./README.md

Execution - command with margins and figures

pandoc -V geometry:"top=2cm, bottom=1.5cm, left=2cm, right=2cm" -f markdown-implicit_figures -o ./README.pdf ./README.md

Execution - command with margins and figures and colored links

pandoc -V linkcolor:blue -V urlcolor:blue -V geometry:"top=2cm, bottom=1.5cm, left=2cm, right=2cm" -f markdown-implicit_figures+autolink_bare_uris -o ./README.pdf ./README.md

note: "autolink_bare_uris" is needed to color in the links (at least with pandoc 3.1.11.1 on Windows).

For more details and examples, see this excellent Medium article at https://pavolkutaj.medium.com/markdown-to-pdf-with-pandoc-and-miktex-58b578cedf4b

Linting Python code using ruff or flake8: simple, fast, free and no false positives

2024-01-19T15:28:00.006+00:00

Linting Python code using ruff or flake8: fast, free and no false positives

Static code analysis can help automatically find minor coding issues and style deviations. This helps especially if several people are collaborating on the same code.

With a dynamically typed language such as Python, since there is no build step, it can be helpful to have some static code analysis in place, to for example catch silly mistakes with indentation, or more serious mistakes such as incorrect exception handling.

Linting can be though of as a kind of lightweight form of static code analysis, looking for code style and layout issues. Because Python uses indenting to communicate control flow and scope (rather than the brackets used by many C++ or C# styled languages including Java), then automatically detecting such spacing issues can be critical.

When selecting a Python linter, there are many options to chose from. I like to use flake8 as it is simple, reliable, fast and has very low rate of false positives. It can still detect more complex issues such as incorrect exception handling.

ruff - linter and formatter

A new and even faster option: ruff - very fast and also formats the code, and its default settings seem to be more practical than flake8.

Installation

To install ruff (via pip):

python -m pip install ruff --quiet

Execution

To lint:

ruff check src

To format:

ruff format src

To configure:

Either add sections to pyproject.toml if you have one, OR add a file ruff.toml.

See the ruff docs for details.

flake8 - simple linter

Installation

To install flake8 (via pip):

python -m pip install flake8 --quiet

Execution

To run, assuming that source code is under 'src' folder, use a command like:

python -m flake8 ./src

For a more complete example with some disabled rules, see the following bash script:

# so future errors halt the script.
set -e

echo Linting ...

# Disable less useful errors
# - ref https://www.flake8rules.com/
#
# E302 Expected 2 blank lines
# E305 Expected 2 blank lines after end of function or class
# E501 Line too long (82 > 79 characters)
FLAKE8_TO_IGNORE=E302,E305,E501

python -m flake8 --extend-ignore $FLAKE8_TO_IGNORE ./src

echo [done]

The same code is also available as a gist.

For more details about flake8, see:

- quickstart: https://flake8.pycqa.org/en/latest/index.html#quickstart

- rules: https://www.flake8rules.com/

Alternatives to flake8 or ruff

For other options to lint Python code (including code style) see the following articles:

- https://codilime.com/blog/python-code-quality-linters/

- https://inventwithpython.com/blog/2022/11/19/python-linter-comparison-2022-pylint-vs-pyflakes-vs-flake8-vs-autopep8-vs-bandit-vs-prospector-vs-pylama-vs-pyroma-vs-black-vs-mypy-vs-radon-vs-mccabe/

Self-host your photos on Amazon CloudFront and S3: Cheap, fast and simple [OSS noobgallery]

2024-01-19T15:02:00.003+00:00

noobgallery is an open source tool at https://github.com/brendannee/noobgallery which I started using after looking at other options that were either too complicated, or just did not create the kind of photo-first experience I desired.

I've used noobgallery to completely rebuild my WordPress based site that was heavy, slow, hard to back up, and also becoming expensive.

noobgallery is a Node.js based project to generate a static photo gallery powered by lightgallery.js. The generated site is automatically deployed to Amazon S3, which allows for a huge amount of cheap storage (around $0.02/GB per month).

Features include:

simple photo gallery creation: just create a folder for each gallery, and copy your photos. noobgallery takes care of the rest.
responsive layout, supports mobile and desktop
slideshow and zoom in/out
gallery summary & tags (via optional gallery.json for each gallery)
bread crumbs to navigate multple gallery levels
about page
error page
does not upload original images
add optional copyright watermark
add optional fotomoto store links
Adding CloudFront is quite trivial, and with Route 53 you get a free HTTPS certificate.

The CloudFront CDN (Content Delivery Network) and the elegant HTML and JavaScript output of noobgallery produce a photo site that is cross-device and is ridiculously fast to load, world wide.

So if you are looking for a cheap and simple way to host your own photo site, do consider noobgallery.

As a shameless plug, here is my own photo website, automatically generated via noobgallery:

https://everydayphotos.net/

disclosure: I did not create noobgallery, but I did contribute a couple of features.

Sharing Python code via pypi packages - example: common Python utils bundled as 'cornsnake'

2024-01-19T14:48:00.005+00:00

To avoid copy-paste hell, (duplicate code is generally regarded to be correlated with bugs) - Python code can be shared via packages.

There is a really nice tutorial here, with just enough detail to walk you through building and publishing your own package:

https://realpython.com/pypi-publish-python-package/

For an example with scripts to automatically bump versions and re-deploy, see:

https://github.com/mrseanryan/cornsnake

cornsnake is a small Python package published via pypi - it contains higher-level wrappers for common Python tasks:

- for working with files, git, ZIP, lists, processes, dates and times.

anti patterns codin'

Fixing or Repairing Windows Installation and Checking Disks for Errors

REST API guidelines from Microsoft

REST API guidelines from Microsoft

Organize the API design around resources

Define API operations in terms of HTTP methods

Source

Pydantic: Dynamically creating a model, with field types, examples and descriptions.

Pydantic: Dynamically creating a model, with field types, examples and descriptions.

Python: Convert a base64 encoded PDF file to a list of PIL (pillow) Images

Problem: poetry does not pick up the current 'local' python (set via pyenv)

Problem: poetry does not pick up the current 'local' python (set via pyenv)

Solution:

A starter template for a Python Command Line program

A starter template for a Python Command Line program

How to read and write a Pydantic class to/from a JSON file.

How to read and write a Pydantic class to/from a JSON file.

Python Debugging Tips

Adding custom 'summary' charts to MLflow: simple bar chart and 'grouped' bar chart

Simple Bar Chart

Grouped Bar Chart

Summary

Show the source code of a Python module (also in Jupyter)

How to show the source code of Python module (also in Jupyter)

Articles about tests involving 3rd party services - End-to-end instead of Integration or Contract tests

Exporting a Jupyter notebook without the input code

World maps in Python - free map resources

Avoiding SettingsWithCopyWarning from pandas when adding a column as function of other columns

Another approach: use df.assign()

Expanding columns and rows in pandas results in Jupyter notebook

Expanding column widths

Expanding row counts

Converting a Python array to a pandas dataframe [multiplying data]

Ubuntu apt (and apt-get) tips

Comparing Costs of Cloud Providers - vantage.sh

Structured Logging (Semantic Logging) in Python and C#

Why Structured logging:

Structured logging in C#:

Further reading - C#

Structured logging in Python:

Further reading - Python

Scanning Python code for security issues via Snyk: the ‘industrial strength’ option

Scanning Python code for security issues via Snyk: the ‘industrial strength’ option

Alternatives to Snyk

AWS Translate via the AWS CLI command line - error: 'charmap' codec can't encode characters

Generate PDF from markdown via pandoc

Installation

Execution - basic command

Execution - command with margins and figures

Execution - command with margins and figures and colored links

Linting Python code using ruff or flake8: simple, fast, free and no false positives

Linting Python code using ruff or flake8: fast, free and no false positives

ruff - linter and formatter

Installation

Execution

flake8 - simple linter

Installation

Execution

Alternatives to flake8 or ruff

Self-host your photos on Amazon CloudFront and S3: Cheap, fast and simple [OSS noobgallery]

Sharing Python code via pypi packages - example: common Python utils bundled as 'cornsnake'