"FosforiVerdi"

Leveraging the Regularizing Effect of Mixing Industrial and Open Source Data to Prevent Overfitting of LLM Fine Tuning

2024-07-09T17:15:00.001+02:00

Leveraging the Regularizing Effect of Mixing Industrial and Open Source Data to Prevent Overfitting of LLM Fine Tuning

Mohamed Salah Jebali , Anna Valanzano , Giacomo Veneri ,Malathi Murugesan, Giovanni De Magistris

Language models have demonstrated important advancements across various natural language processing (NLP) tasks. However, the availability of high-quality and domain-specific data remains a challenge for training these models, particularly in industry-specific applications. In this paper, we propose a methodology to fine-tune a large language model (LLM) using a mixture of private company data and open-source data. \\Our empirical investigation reveals that combining private and open-source data during the fine-tuning process leads to superior performance, mitigating the risk of overfitting that can occur when training solely on narrow, domain-specific datasets. We observed that incorporating open-source data alongside the private data helps to reduce the distribution shift between the source and target data, effectively acting as a regularizer and enhancing the model's ability to generalize.\\Furthermore, we compare the divergence between the private and open-source datasets with the test loss of the fine-tuned model. Our results suggest a correlation between reduced data divergence and improved model performance, indicating that carefully selecting and curating the dataset mixture can be a crucial step in preventing overfitting and ensuring the model's effective adaptation to industry-specific use cases.

This study provides a practical solution for industry-specific adaptation of LLMs, demonstrating how the strategic blending of private and open-source data can unlock the full potential of these models while addressing critical concerns around data privacy and model reliability in real-world applications.

Codellama for SQL

2024-03-19T21:01:00.009+01:00

In this simple exercise we are going to use CodeLLama to interact with Postgresql DB using natural language.

Prepare Database

Open PostgreSQL DB and build the following schema:

CREATE TABLE customers
(
   cust_id                  INT         NOT NULL PRIMARY KEY,
   cust_first_name          VARCHAR(20)   NOT NULL,
   cust_last_name           VARCHAR(40)   NOT NULL,
   cust_state               VARCHAR(40)   NOT NULL 
);

CREATE TABLE products
(
   prod_id                 INT        NOT NULL PRIMARY KEY,
   prod_name               VARCHAR(50)     NOT NULL,
   prod_desc               VARCHAR(4000)   NOT NULL,
   prod_subcategory        VARCHAR(50)     NOT NULL
);

CREATE TABLE orders
(
   ord_id                 INT        NOT NULL PRIMARY KEY,
   ord_amount             INT        NOT NULL,
   ord_price              INT        NOT NULL,
   ord_prod_id            INT        NOT NULL,
   ord_cust_id            INT        NOT NULL,
   ord_date               DATE       NOT NULL,
   
   CONSTRAINT ord_cust_fk
      FOREIGN KEY (ord_cust_id) REFERENCES customers (cust_id),
    CONSTRAINT ord_prod_fk
      FOREIGN KEY (ord_prod_id) REFERENCES products (prod_id)
);

then populate the db:

COPY public.customers (cust_id, cust_first_name, cust_last_name, cust_state) FROM stdin;
1	giacomo	veneri	Italy
2	mario	rossi	Italy
3	mike	do	USA
\.


COPY public.orders (ord_id, ord_amount, ord_price, ord_prod_id, ord_cust_id, ord_date) FROM stdin;
1	1	1000	1	1	2024-03-15
2	1	1000	1	2	2024-03-15
3	1	100	2	1	2024-03-15
4	2	1000	1	3	2024-03-15
5	1	30	3	2	2024-03-16
\.

COPY public.products (prod_id, prod_name, prod_desc, prod_subcategory) FROM stdin;
1	Computer PC	PC	electronic
2	Tablet	Tablet	electronic
3	My Book	My Title	book
\.

COPY public.customers (cust_id, cust_first_name, cust_last_name, cust_state) FROM stdin;
1	giacomo	veneri	Italy
2	mario	rossi	Italy
3	mike	do	USA
\.


COPY public.orders (ord_id, ord_amount, ord_price, ord_prod_id, ord_cust_id, ord_date) FROM stdin;
1	1	1000	1	1	2024-03-15
2	1	1000	1	2	2024-03-15
3	1	100	2	1	2024-03-15
4	2	1000	1	3	2024-03-15
5	1	30	3	2	2024-03-16
\.

COPY public.products (prod_id, prod_name, prod_desc, prod_subcategory) FROM stdin;
1	Computer PC	PC	electronic
2	Tablet	Tablet	electronic
3	My Book	My Title	book
\.

Install the Packages

pip install bitsandbytes accelerate simpletransformers 
pip uninstall -y transformer-engine
pip install psycopg2-binary SQLAlchemy

Instantiate the model

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
 
model_id =  "codellama/CodeLlama-7b-Instruct-hf"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
 
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")
 
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
eos_token_id = tokenizer.eos_token_id

Now we can start connection to the local db

import psycopg2 as pg
engine = pg.connect("dbname='LLM_TEST' user='postgres' host='localhost' port='8432' password='password'")

Prepare Prompting

schema = """
CREATE TABLE customers
(
   cust_id                  INT         NOT NULL PRIMARY KEY,
   cust_first_name          VARCHAR(20)   NOT NULL,
   cust_last_name           VARCHAR(40)   NOT NULL,
   cust_state               VARCHAR(40)   NOT NULL 
);

CREATE TABLE products
(
   prod_id                 INT        NOT NULL PRIMARY KEY,
   prod_name               VARCHAR(50)     NOT NULL,
   prod_desc               VARCHAR(4000)   NOT NULL,
   prod_subcategory        VARCHAR(50)     NOT NULL
);

CREATE TABLE orders
(
   ord_id                 INT        NOT NULL PRIMARY KEY,
   ord_amount             INT        NOT NULL,
   ord_price              INT        NOT NULL,
   ord_prod_id            INT        NOT NULL,
   ord_cust_id            INT        NOT NULL,
   ord_date               DATE       NOT NULL,
   
   CONSTRAINT ord_cust_fk
      FOREIGN KEY (ord_cust_id) REFERENCES customers (cust_id),
    CONSTRAINT ord_prod_fk
      FOREIGN KEY (ord_prod_id) REFERENCES products (prod_id)
);
"""

import sqlparse
def query(sql,engine=engine):
    print(f'"SQL QUERY:\n {sql}"')
    return pd.read_sql(sql, con=engine)


def ask(question, schema=schema, model=model, tokenizer=tokenizer):

    sql_example = "SELECT * , orders.ord_price AS spent FROM orders INNER JOIN products ON orders.ord_prod_id = products.prod_id INNER JOIN customers ON orders.ord_cust_id = customers.cust_id"
    
    eval_prompt = f"""### Task
    # Generate a Postgresql SQL query to answer the following question:
    # `{question}

    # Please wrap your code answer using ```sql:

    # ### Database Schema
    # This query will run on a database whose schema is represented in this string:
    # Don't use joins for this schema and if all columns are required give the (*) notation.

    # `{schema}`

    # An example of the SQL would be `{sql_example}`

    # ### SQL
    # Given the database schema, here is the SQL query that answers `{question}`:
    # ```sql
    # """
    
    #eval_prompt = f"[INST] Write SQL query to answer the following question given the database schema. Please wrap your code answer using ```sql: Schema: {schema} Question: {question} [/INST] Here is the SQL query to answer to the question: {question}: ```sql "


    model_input = tokenizer(eval_prompt, return_tensors="pt").to("cuda")
    
    #model.eval()

    outputs = tokenizer.batch_decode(model.generate(**model_input,
                                                    eos_token_id=eos_token_id,
                                                    pad_token_id=eos_token_id,
                                                    max_new_tokens=100,
                                                    do_sample=False, num_beams=1), 
                                     skip_special_tokens=True)

    ret =  sqlparse.format(outputs[0].split("```")[-2], reindent=True).replace('#','').replace("sql ","")
 
    return ret,outputs

def ask_db(question):
    try:
        sql, outputs = ask(question)
        return query(sql)
    except Exception as e:
        print('error', e)
        print("*"*10)
        print(sql)
        print("*"*10)
        print(outputs)

Asking data

Now we can ask our data in natural language:

ask_db(input())

Can you tell me about the total spent by giacomo?

The output

"SQL QUERY:
   SELECT SUM(ord_price) AS spent
   FROM orders
   INNER JOIN customers ON orders.ord_cust_id = customers.cust_id
   WHERE customers.cust_first_name = 'giacomo'
 "

spent

More examples

Can you tell me about the orders done by customer giacomo?

Can you tell me about the total amount ordered by customer giacomo?
Can you tell me about the orders done by customer of italy?
Can you tell me about the orders done the last week by customer giacomo?
Can you tell me about the products of subcategory electronic?
Can you tell me about the total spent by giacomo?

Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks in Scientific Computing

2024-01-31T21:53:00.005+01:00

From the review https://arxiv.org/pdf/2211.07377.pdf

In the last decade, breakthroughs in DL and computing power have enabled the use of DL in a broad variety of scientific computing, especially in

fluid mechanics,
solid mechanics, and
materials science,

albeit at the cost of accuracy and loss of generality.

These data-driven methods are routinely applied to fulfill one of the following goals:

accelerate direct numerical simulations using surrogate modeling
accelerate adjoint sensitivity analysis
accelerate probabilistic programming
accelerate inverse problems

PGNN

In Physics-Guided NN-based models, off-the-shelf supervised DL techniques are used to construct surrogate mappings. (see also https://towardsdatascience.com/physics-guided-neural-networks-pgnns-8fe9dbad9414)

The idea is to manage physical constraints in the loss.

or you can manage with Physics intermediate variable

PINNs

Physics-Informed NNs have been introduced to perform supervised learning tasks while obeying given laws of physics in the form of general non-linear differential equations

PeNN

Physics-Encoded neural networks (PeNN) based models leverage advanced architectures to address issues with data sparsity and the lack of generalization encountered by both PgNNs and PiNNs models.

Neural Operator

Moreover a neural operator can be combined with PiNN and PeNN methods to train a model that can learn complex non-linearity in physical systems with extremely high generalization accuracy.

Train LLAMA-2 on a Small GPU

2024-01-11T07:06:00.002+01:00

Training a LLM model is hard without GPU but we can use LORA and QLORA

4-bit quantization via QLoRA allows efficient finetuning of huge LLM models on consumer hardware while retaining high performance. This improves accessibility and usability for real-world applications.

QLoRA quantizes a pre-trained language model to 4 bits and freezes the parameters.

During fine-tuning, gradients are backpropagated through the frozen 4-bit quantized model into only the Low-Rank Adapter layers.

The 4-bit quantization does not hurt model performance.

Quantization

Quantization techniques reduces memory and computational costs by representing weights and activations with lower-precision data types like 8-bit integers (int8).

In this post we use 4bit quantization

pip install bitsandbytes

...

quant_config = BitsAndBytesConfig(
	load_in_4bit=True,
	bnb_4bit_quant_type="nf4",
	bnb_4bit_compute_dtype=getattr(torch, "float16"),
	bnb_4bit_use_double_quant=False,
)

Training parameters

Using LoraConfig

pip install peft_params

...

peft_params = LoraConfig(
	lora_alpha=16,
	lora_dropout=0.1,
	r=64,
	bias="none",
	task_type="CAUSAL_LM",
)


training_params = TrainingArguments(
	output_dir="./results",
	num_train_epochs=1,
	per_device_train_batch_size=4,
	gradient_accumulation_steps=1,
	optim="paged_adamw_32bit",
	save_steps=25,
	logging_steps=25,
	learning_rate=2e-4,
	weight_decay=0.001,
	fp16=False,
	bf16=False,
	max_grad_norm=0.3,
	max_steps=-1,
	warmup_ratio=0.03,
	group_by_length=True,
	lr_scheduler_type="constant"
)

trainer = SFTTrainer(
	model=model,
	train_dataset=dataset,
	peft_config=peft_params,
	dataset_text_field="text",
	max_seq_length=None,
	tokenizer=tokenizer,
	args=training_params,
	packing=False,
)

Prepare Docker

Below how to configure the docker to run on RTX A3000

Base Image

FROM nvidia/cuda:12.1.1-base-ubuntu20.04

Set environment variables

Install system dependencies

RUN apt-get update
RUN apt-get install -y \
		git \
		python3-pip \
		python3-dev \
		python3-opencv \
		libglib2.0-0

Install PyTorch and torchvision

RUN pip3 install torch torchvision torchaudio -f https://download.pytorch.org/whl/cu121

Install any python packages you need

COPY requirements.txt requirements.txt
RUN python3 -m pip install notebook accelerate peft bitsandbytes transformers trl tensorboard

Full code

full code is there

https://github.com/venergiac/training_LLAMA-2_QLORA

Build docker:

docker compose build

Start docker:

docker compose run

Open the jupyter and check it with nvidia-smi.

Towards General AI

2024-01-02T13:35:00.003+01:00

Recent advancements in Large Language Models (LLMs) have demonstrated a ability for reasoning. Is this the basis for General AI? General AI (AGI) is known as human-level AI or general intelligent action. AGI is able to learn tasks from one domain and apply to another domain; so AGI is the AI able to perform tasks in a very similar way to humans.

In 2020 we predicted that AGI would be reached in 2040; we have now moved forward to 2030.

Nvidia CEO Jensen Huang says AI will be ‘fairly competitive’ with humans in 5 years

The road is still foggy (and perhaps we are being optimistic), but some signs are showing us the way.

Human vs AI

Let's retrace the road. Human is a collection of complex capabilities:

The ability to remember,

to perceive the environment and interact with it,
the ability to learn,
the ability to generalize and to reason in abstract and symbolic way
the self-awareness
and, last but not least, the ability to judge ethically own actions.

To emulate these functions, several strategies have been developed: in 1950 we deceived ourselves that we understood the visual mechanisms and we created the perceptron (1958), the later convolutional filters. In 1980 we defined the decision trees to learn from data. Only thanks, however, to the growing computational capacity, we obtained the first results (LeNet and CNN) in 2010.

Deep Learning and Transformers architecture (2017) led us to Large Language Models (LLM, 2020): models with over 1 billion parameters.

Large Language Model

A feature predominantly seen in models with over 100 billion parameters, is the ability of reasoning (considered an emergent ability). While opinions differ on whether LLMs possess reasoning abilities, it's becoming evident that LLMs can be equipped to deal with complex problems (similar to human).

Indeed, through sophisticated prompting strategies (eg. Chain-of-Thought Prompting, Tree of Thoughts, ....) or Symbolic Modules it is possible to achieve complex tasks.

So, what is the next step to achieve the AGI?

Multimodality

The secret ingredient seems to be (obviously) multimodality; that is, the ability to integrate and process audio, video and sound content in an integrated manner. It's not just adding something but also improving.

Gemini (Google), Ferret (Apple) and GPT-4+ (OpenAi) and other open source (eg. Hugging Face, LangChain, ...) tools are going in this direction.

Moreover, cross-modal alignment is another interesting tool: Contrastive Language Image Pretraining (CLIP) utilizes image captioning task to align image and text modalities. Contrastive Language Audio Pretraining aligns audio and text.

So, what is the next step to achieve AGI?

1 more Multimodality

There is, however, a fourth element to integrate: not only vision, text and audio, but also the ability to act and interact with an environment.

Tools like Reinforcement Learning or IoT or Robotic or simple API can provide to AI the ability to take a further step.

Super AI

Artificial Super Intelligence (ASI) is defined as a form of AI capable of surpassing human intelligence. ASI is expected by 2050, but, first, we should fix some issues such as ethics and self-awareness (and maybe leveraging on quantum computing).

Conclusion

So is it 2024 when we will find the road to AGI?

Hands-On Industrial Internet of Things - my best book - thanks to all

2023-12-09T13:17:00.006+01:00

Achievement of my book Hands-On Industrial Internet of Things

7000 copies, 40+ scientific citations, 4.4 on amazon

2019 Best Book

It's AI generating generative AI commissions?

2023-12-06T09:20:00.006+01:00

It's AI generating generative AI commissions?

BiomedLM - Domain-Specific Large Language Model for Biomedical Text

2023-11-15T20:33:00.004+01:00

Domain-Specific Large Language Model for Biomedical Text

BioMedLM

BioMedLM 2.7B is new language model trained exclusively on biomedical abstracts and papers.

import torch

from transformers import GPT2LMHeadModel, GPT2Tokenizer

device = torch.device("cuda")

tokenizer = GPT2Tokenizer.from_pretrained("stanford-crfm/pubmed_gpt_tokenizer")

model = GPT2LMHeadModel.from_pretrained("stanford-crfm/pubmedgpt").to(device)

and now ask your question about mastocitosis

input_ids = tokenizer.encode(
    "How to treat mastocytosis in 10 words ?", return_tensors="pt"
).to(device)

sample_output = model.generate(input_ids, do_sample=True, max_length=200, top_k=50)

print("Output:\n" + 100 * "-")
print(tokenizer.decode(sample_output[0], skip_special_tokens=True))

the response

How to treat mastocytosis in 10 words?
Mastocytosis, a heterogeneous group of diseases characterized by an abnormal proliferation of tissue mast cells, is rarely seen in the pediatric population. A clinical approach to the diagnosis and treatment of mastocytosis is provided. The treatment of mastocytosis is challenging and has significant limitations. This review provides clinicians and patients with a concise, organized, and practical approach to the diagnosis and treatment of mastocytosis. Recent advances in the management of mastocytosis include biologic agents (pegylated interferon, omalizumab, omaliraptor), a bone marrow transplantation protocol, and novel chemotherapeutic agents. Despite these advances in treatment, the goal of disease control remains unmet for many patients.

NASSCOM Award - NOx estimator for a gas turbine hydrogen-fueled burner

2023-10-15T10:36:00.003+02:00

Development of a surrogate model for NOx emissions for 100% hydrogen-fueled burners, leveraging deep learning methods and some advanced techniques for feature importance. Allows combustion design engineers to evaluate NOx emission levels, given the operating parameters of the combustion chamber, at an early, design stage, so to also reduce design rework and testing activity. Feature importance helps identifying the parameters to leverage to minimize emissions.

Use SHAPELY and CATBOOST to debug your model

2023-09-03T20:45:00.010+02:00

Catboost is a boosting method accepting data with categorical features and text; for tabular data is very efficient; sometime, however, using industrial dataset we can face with overfitting issue and you want to debug using feature importance or better shapely value.

Build a model

Just import

import catboost
import pandas as pd
import numpy as np
import os

Open the dataset

We use the Womens Clothing E-Commerce Reviews.csv
Dataset is available there https://www.kaggle.com/datasets/nicapotato/womens-ecommerce-clothing-reviews

df = pd.read_csv('Womens Clothing E-Commerce Reviews.csv', delimiter=',', nrows = nRowsRead)
df.dataframeName = 'Womens Clothing E-Commerce Reviews.csv'

Build the model

CAT_FEATURES=['Division Name','Department Name','Class Name']
TEXT_FEATURES=['Title', 'Review Text']
NUM_FEATURES=['Age']

X_COLUMNS = NUM_FEATURES + CAT_FEATURES + TEXT_FEATURES
Y_COLUMN = ['Rating']

df[CAT_FEATURES + TEXT_FEATURES] = df[CAT_FEATURES + TEXT_FEATURES].astype(str)


from sklearn.model_selection import train_test_split
df_train, df_test = train_test_split(df, test_size=0.3, random_state=42)



model = catboost.CatBoostRegressor(cat_features=CAT_FEATURES, 
                          text_features=TEXT_FEATURES,
                          iterations=1000,
                          early_stopping_rounds=100)

# Fit model
model.fit(df_train[X_COLUMNS], df_train[Y_COLUMN])

Eval model

model.score(df_test[X_COLUMNS], df_test[Y_COLUMN])

0.2854713235691765

Debug with shapley

import shap
from catboost import *
shap.initjs()

X = df_test[X_COLUMNS]
y = df_test[Y_COLUMN]

pool = Pool(X, y, cat_features=CAT_FEATURES, text_features=TEXT_FEATURES)


explainer = shap.TreeExplainer(model)
shap_values = explainer(pool)

# summarize the effects of all the features
shap.summary_plot(shap_values, X)

shap.plots.bar(shap_values)

For source : https://www.kaggle.com/code/giacomoveneri1/notebooka6d3db12e2

Combining Thermodynamics-based Model of the Centrifugal Compressors and Active Machine Learning for Enhanced Industrial Design Optimization

2023-09-02T11:23:00.006+02:00

Combining Thermodynamics-based Model of the Centrifugal Compressors and Active Machine Learning for Enhanced Industrial Design Optimization - S Ghiasi, G Pazzi, C Del Grosso, G De Magistris, G Veneri - 1st workshop on Synergy of Scientific and Machine Learning Modeling, SynS - 2023

The design process of centrifugal compressors requires applying an optimization process which is computationally expensive due to complex analytical equations underlying the compressor’s dynamical equations. Although the regression surrogate models could drastically reduce the computational cost of such a process, the major challenge is the scarcity of data for training the surrogate model. Aiming to strategically exploit the labeled samples, we propose the Active Comp Design framework in which we combine a thermodynamics-based compressor model (i.e., our internal software for compressor design) and Gaussian Process- based surrogate model within a deployable Active Learning (AL) setting. We first conduct experiments in an offline setting and further, extend it to an online AL framework where a real-time interaction with the thermodynamics based compressor’s model allows the deployment in production. Active Comp Design shows a significant performance improvement in surrogate modeling by leveraging on uncertainty-based query function of samples within the AL framework with respect to the random selection of data points. Moreover, our framework in production has reduced the total computational time of compressor’s design optimization to around 46% faster than relying on the internal thermodynamics based simulator, achieving the same performance.

The Algorithm

Results

Conclusions

The benefits of combining active ML methods with physical models underlying compressor’s dynamics are large for design optimization applications including faster computations and more accurate design solutions. However, the tradeoff between performance and computational power has to be carefully evaluated for the specific design application. Moreover (see also (Pardakhti et al., 2021)), the adoption of AL methods poses significant challenges in many practical applications, such as lack of data, discontinuous space of exploration and measurement error. The results obtained from offline and online ActiveCompDesign show that integrating ML in compressor’s simulators is viable for production ready application on the energy sector. Indeed, our framework is currently working in a production environment. For future research we will expand this framework for wider design optimization applications and improve the monitoring of the deployed model.

See also github artificial intelligence lab baker hughes: https://github.com/AILAB-bh/AILAB-bh

Sensor Virtualization for Anomaly Detection of Turbo-Machinery Sensors-An Industrial Application

2023-07-21T10:40:00.004+02:00

Sensor Virtualization for Anomaly Detection of Turbo-Machinery Sensors - An Industrial Application

Sachin Shetty, Valentina Gori, Gianni Bagni and Giacomo Veneri

We apply a Granger causality and auto-correlation analysis to train a recurrent neural network (RNN) that acts as a virtual sensor model. These models can be used to check the status of several hundreds of sensors during turbo-machinery units’ operation. Checking the health of each sensor is a time-consuming activity. Training a supervised algorithm is not feasible because we do not know all the failure modes that the sensors can undergo. We use a semi-supervised approach and train an RNN (LSTM) on non-anomalous data to build a virtual sensor using other sensors as regressors. We use the Granger causality test to identify the set of input sensors for a given target sensor. Moreover, we look at the auto-correlation function (ACF) to understand the temporal dependency in data. We then compare the predicted signal vs. the real one to raise (in case) an anomaly in real time. Results report 96% precision and 100% recall.

Methods

We used the auto-correlation to find the temporal relations in both input sensors and target sensor to obtain the best ``window size'' to train the LSTM model.

Results

Conclusions

In this work, we presented a real industrial application of sensor anomaly detection in the domain of energy and turbomachinery. We applied a semi-supervised deep learning technique, which can be used to perform anomaly detection in an industrial context. In particular, we applied anomaly detection to turbo-machinery units by training a virtual sensor model for a given sensor. We first selected input features through Granger causality and leveraged auto-correlation and subject matter expertise to identify the best window size for the recurrent neural network chosen (LSTM).

This method can be scaled and extended to almost all the sensors installed on the unit, for a complete sensor anomaly detection system.

Furthermore, once the model has been trained for a single sensor, we can later retrain the model using data collected over time, with a continual learning approach so that the algorithm is able to also take into account data-shift phenomena.

Cite as:

@inproceedings{shetty2023,
  author =       {Sachin Shetty and Valentina Gori and Gianni Bagni and Giacomo Veneri},
  title =        {Sensor virtualization for anomaly detection of turbo-machinery sensors - An industrial application},
  booktitle =    "Proceedings of the 9th International conference on Time Series and Forecasting - ITISE2023",
  publisher =    {MDPI}
  venue =        {Gran Canaria, Spain},
  month =        jul,
  year =         {2023}
}

Tensorflow Custom Functions - cheat sheet

2023-07-02T09:08:00.001+02:00

Building custom function with keras and tensorflow 2 is, sometime, error prone.

Below a collection of tensorflow numpy mapping

Refer to : https://www.tensorflow.org/guide/function

Losses Good Practice from * to ***

Huber Loss Simple (*)

def my_huber_loss(y_true, y_pred):
    threshold = 1
    error = y_true - y_pred
    is_small_error = K.abs(error) <= threshold
    small_error_loss = K.square(error) / 2
    big_error_loss = threshold * (K.abs(error) - (0.5 * threshold))
    return K.where(is_small_error, small_error_loss, big_error_loss)

Huber Loss with parameter (**)

import tensorflow as tf
#wrapper function which accepts the threshold parameter
def my_huber_loss_with_threshold(threshold):
   def my_huber_loss(y_true, y_pred):   
  
       error = y_true - y_pred     
       is_small_error = K.abs(error) <= threshold     
       small_error_loss = K.square(error) / 2     
       big_error_loss = threshold * (K.abs(error) - (0.5 * threshold))
       return K.where(is_small_error, small_error_loss, big_error_loss)
    return my_huber_loss

Contrastive Loss with parameter (**)

def contrastive_loss_with_margin(margin):
    def contrastive_loss(y_true, y_pred):
        
        square_pred = K.square(y_pred)
        margin_square = K.square(K.maximum(margin - y_pred, 0))
        return K.mean(y_true * square_pred + (1 - y_true) * margin_square)
    return contrastive_loss

Huber Loss Class (***)

from tensorflow.keras.losses import Loss

class MyHuberLoss(Loss):
  
    # initialize instance attributes
    def __init__(self, threshold=1):
        super().__init__()
        self.threshold = threshold

    # compute loss
    def call(self, y_true, y_pred):
        error = y_true - y_pred
        is_small_error = K.abs(error) <= self.threshold
        small_error_loss = K.square(error) / 2
        big_error_loss = self.threshold * (K.abs(error) - (0.5 * self.threshold))
        return K.where(is_small_error, small_error_loss, big_error_loss)

Hands-On Industrial Internet of Things - 2023

2023-01-03T19:45:00.005+01:00

Let us say THANKS (because we have no other way) to all readers and reviewers who, after 4 years,

confirmed 4.4 stars to our book

"Hands-On Industrial Internet of Things"

with

~4 copies (50% paperback 50% digital) sold every day, > 5500 copies sold,
34 scientific citations,
nominated by Book Authority in 2019 one of the "TOP 5 best" IOT book,
62 reviews,
available on 12 book platforms (Oreilly, Amazon , IBS, Google, .... and obviously Packt ).

Building a custom model in scikit learn

2022-10-20T21:33:00.007+02:00

Scikit learn is one of the most interesting Machine Learning package for python never seen. The current version is the 1.1.2 .

The base api of scikit-learn is very clean and well developed:

Estimator for fitting
Predictor to predict new data
Transformer to transform data
Model just for score

If you want to develop your custom model you need to implement tow interfaces

```
BaseEstimator
```
```
RegressorMixin or ClassifierMixin
```

So below is a simple classifier that look for the closest x an return the y. In this case it is valid for classification and regression:

class BaseSearcher(BaseEstimator, ClassifierMixin, RegressorMixin):

     def __init__(self, demo_param='demo'):
         self.demo_param = demo_param

     def fit(self, X, y):

         # Check that X and y have correct shape
         X, y = check_X_y(X, y)
         # Store the classes seen during fit
         self.classes_ = unique_labels(y)

         self.X_ = X
         self.y_ = y
         # Return the classifier
         return self

     def predict(self, X):

         # Check if fit has been called
         check_is_fitted(self)

         # Input validation
         X = check_array(X)

         closest = np.argmin(euclidean_distances(X, self.X_), axis=1)
         return self.y_[closest]

Document Layout Analysis with Variational Autoencoders: An Industrial Application

2022-09-29T08:42:00.005+02:00

Document Layout Analysis with Variational Autoencoders: An Industrial Application

Ali Youssef, Gabriele Valvano & Giacomo Veneri
Part of the Lecture Notes in Computer Science book series (LNAI,volume 13515)

We present a novel method for Document Layout Analysis that detects documents that are not compliant with a given template. The major challenge we solve is dealing with a highly unbalanced dataset with only a few, hard-to-distinguish, non-compliant documents. Our model learns to detect inadequate documents based on localised non-compliant characteristics, including stamps, handwritten text, and misplaced signatures. Nevertheless, the model must not report documents containing other artefacts such as amendments or notes, which we deem acceptable. We address these challenges via generative modelling, using anomaly detection techniques to validate document layout. In particular, we first let the model learn the compliant document distribution. Then, we detect and report out-of-distribution samples for their automated rejection. In the paper, we investigate and compare two major approaches to anomaly detection: 1) classifying anomalies as those samples that cannot be accurately generated by the model; and 2) detecting samples whose mapping to a known proxy distribution is not possible. Both methods can be trained without annotations and obtain a classification accuracy of ∼∼90% on real-world documents, outperforming alternative supervised solutions.

We used CNN-VAE and CNN for anomaly detection.

Architectures:

We used 2 architectures: VAE + SSIM and VAE+Classifier

Results:

Cite as:

Youssef, A., Valvano, G., Veneri, G. (2022). Document Layout Analysis with Variational Autoencoders: An Industrial Application. In: Ceci, M., Flesca, S., Masciari, E., Manco, G., Raś, Z.W. (eds) Foundations of Intelligent Systems. ISMIS 2022. Lecture Notes in Computer Science(), vol 13515. Springer, Cham. https://doi.org/10.1007/978-3-031-16564-1_46

Presenting at WCCI2022 my paper

2022-08-08T11:27:00.008+02:00

Continual Learning for anomaly detection on turbomachinery prototypes - A real application

Valentina Gori, Giacomo Veneri, Valeria Ballarini

TPS Artificial Intelligence - Baker Hughes

We apply a Recurrent Neural Network (RNN), Kullback-Leibler (KL) divergence and a continual learning approach to check the status of several hundreds of sensors during turbo-machinery prototype testing. Turbo-machinery prototypes can be instrumented with up to thousands of sensors. Therefore, checking the health of each sensor is a time consuming activity. Prototypes are also tested on several different and a-priori unknown operating conditions, so we cannot apply a purely supervised model to detect potential anomalies of sensors and, moreover, we have to take into account a covariate shift because measurements drift continuously day by day. We continuously train a RNN (daily) to build a virtual sensor from other sensors and we compare the predicted signal vs the real signal to raise (in case) an anomaly. Furthermore, KL is used to estimate the overlap between the input distributions available at training time and the ones seen at test time, and thus the confidence level of the prediction. Finally we implement an end-to-end system to automatically train and evaluate the models. The paper presents the system and reports the application to a test campaign of about five hundred sensors.
Publishes at CEC IEEE 2022

Speaker: Giacomo Veneri

Challenge

In the industry specific domain we apply anomaly detection to check the health of the sensors installed on our prototypes of turbo-machines.
A supervised approach is not feasible because anomalous signals can show disparate aspects: expected signals for well-behaved sensors are not available since each prototype is unique and usually tested just once.
Covariate shift issues must be managed, because operating conditions may change severely during test campaign.
Up to few thousand sensors need to be checked.

The process

Main Contribution

Paper explained:

Deep Surrogate of Modular Multi Pump using Active Learning

2022-08-08T11:16:00.003+02:00

Review of: Deep Surrogate of Modular Multi Pump using Active Learning

Malathi Murugesan, Kanika Goyal, Laure Barriere, Maura Pasquotti, Giacomo Veneri, Giovanni De Magistris
Due to the high cost and reliability of sensors, the designers of a pump reduce the needed number of sensors for the estimation of the feasible operating point as much as possible. The major challenge to obtain a good estimation is the low amount of data available. Using this amount of data, the performance of the estimation method is not enough to satisfy the client requests. To solve this problem of scarcity of data, getting high quality data is important to obtain a good estimation. Based on these considerations, we develop an active learning framework for estimating the operating point of a Modular Multi Pump used in energy field. In particular we focus on the estimation of the surge distance. We apply Active learning to estimate the surge distance with minimal dataset. Results report that active learning is a valuable technique also for real application.

Published at ReALML @ ICML2022

https://realworldml.github.io/accepted/

This paper demonstrates that using active learning we can reduce the amount of data to learn a surrogate model.

Attached the poster

Atlassian Jira vs Broadcom Rally

2022-08-08T11:10:00.000+02:00

Comparing Jira vs Rally is very difficult because both offer strong project management features. Jira allows high customisable query language, rather than Rally leverages on (limited) plugin.

Rally leads the agile team, while Jira leaves them free to define their own process; for this reason, Jira requires high customisation but Rally is more limited for Kanban.

Conclusion

Product Owner and SCRUM team loves Rally, but developers prefer Jira
Jira is highly customizable, it can be used in different industries with different needs.
Rally Software is good for sprint management.

...and I prefer Jira ✌

MLOPS vs DEVOPS

2022-02-21T12:37:00.003+01:00

DEV OPS

Continuos Integration
Continuos Delivery

ML OPS

Continuous Training

Hands-On Industrial Internet of Things - Thanks

2022-02-12T19:25:00.008+01:00

Let us say THANKS (because we have no other way) to all readers and reviewers who, after 3 years,

confirmed 4.5 stars to our book

"Hands-On Industrial Internet of Things"

with

~4 copies (40% paperback 60% digital) sold every day,
24 citations ,
nominated by Book Authority in 2019 one of the "TOP 5 best" IOT book
44 reviews

DANNTe: a case study of a turbo-machinery sensor virtualization under domain shift -

2021-12-06T20:28:00.001+01:00

Luca Strazzera, Valentina Gori, Giacomo Veneri

We propose an adversarial learning method to tackle a Domain Adaptation time series regression task (DANNTe). The task concerns the virtualization of a physical sensor of a turbine with aim to build a reliable virtual sensor working on operating conditions not considered during the training phase. Our approach is directly inspired by the need to have a domain-invariant representation of the features to correct the covariate shift present in the data. The learner has access to both a labeled source data and unlabeled target data (Unsupervised DA) and is trained on both, exploiting the minmax game between a task regressor neural network and a domain classifier neural network. Both models share the same feature representation in terms of a feature extractor neural network. This work is based on the work of Ganin et al.; we present an extension suitable to be applied to time series data. The results report a significant improvement in regression performance, compared to the base model trained on the source domain only.

Tensorflow Similarity applied to NLP

2021-10-23T10:34:00.002+02:00

Tensorflow Similarity offers state-of-the-art algorithms for metric learning and all the necessary components to train and serve similarity-based models.

Recent version introduced new losses:

Triplet Loss
PN Loss
Multi Sim Loss
Circle Loss

The basic idea of Tensorflow Similarity i to provide a framework to train similarity model. On internet you can find a lot of example based on MNIST (one more MNIST example 🙈); in this paper we would like to apply TF Similarity to a NLP model based on a custom dataset.

DataSet

We can use IMDB dataset (yes another sentiment analysis dataset 🙈). You can download for free from there: https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews

The IMDB Dataset is a two classes text dataset.

IMDB Example

Let's import the dataset and encode the label to a binary 0,1 vector.

# loading from file
from tqdm import tqdm
import pandas as pd

import numpy as np

PATH_DATA = 'IMDB Dataset.csv'

df = pd.read_csv(PATH_DATA)

target_column='sentiment'
text_column='review'

from sklearn import preprocessing
le = preprocessing.LabelEncoder()

X = df[text_column].astype(str).values
y = le.fit_transform(df[target_column].astype(str).values)

We are working with text and we need to build a layer to vectorize the text.

import tensorflow as tf
from tensorflow.keras.layers import TextVectorization
import string
import re

# Having looked at our data above, we see that the raw text contains HTML break
# tags of the form '<br />'. These tags will not be removed by the default
# standardizer (which doesn't strip HTML). Because of this, we will need to
# create a custom standardization function.
def custom_standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    stripped_html = tf.strings.regex_replace(lowercase, "<br />", " ")
    return tf.strings.regex_replace(
        stripped_html, "[%s]" % re.escape(string.punctuation), ""
    )


# Now that we have our custom standardization, we can instantiate our text
# vectorization layer. We are using this layer to normalize, split, and map
# strings to integers, so we set our 'output_mode' to 'int'.
# Note that we're using the default split function,
# and the custom standardization defined above.
# We also set an explicit maximum sequence length, since the CNNs later in our
# model won't support ragged sequences.
vectorize_layer = TextVectorization(
    standardize=custom_standardization,
    max_tokens=max_features,
    output_mode="int",
    output_sequence_length=sequence_length,
)

# Now that the vocab layer has been created, call `adapt` on a text-only
# dataset to create the vocabulary. You don't have to batch, but for very large
# datasets this means you're not keeping spare copies of the dataset in memory.

vectorize_layer.adapt(X)

The inevitable split and train

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify = y)

Now we can import our dataset on MultiShotMemorySampler. MultiShotMemorySampler is a class provided by tensorflow similarity:

from tensorflow_similarity.samplers import MultiShotMemorySampler  # sample data 

CLASSES = np.unique(y)
NUM_CLASSES = CLASSES.shape[0]
CLASS_PER_BATCH = NUM_CLASSES
EXAMPLE_PER_CLASS = 1 
STEPS_PER_EPOCH = 1000 

sampler_train = MultiShotMemorySampler(X_train, y_train, 
                                 classes_per_batch=CLASS_PER_BATCH, 
                                 examples_per_class_per_batch=EXAMPLE_PER_CLASS,
                                 #class_list=CLASSES, # Only use the first 6 classes for training.
                                 steps_per_epoch=STEPS_PER_EPOCH)

sampler_test = MultiShotMemorySampler(X_test, y_test, 
                                 classes_per_batch=CLASS_PER_BATCH, 
                                 examples_per_class_per_batch=EXAMPLE_PER_CLASS,
                                 #class_list=CLASSES[:NUM_CLASSES], # Only use the first 6 classes for training.
                                 steps_per_epoch=STEPS_PER_EPOCH)

Building Model

This is a very simple model based on Conv1D; you can customize your architecture. Please note it uses our vectorize layer defined before.

from tensorflow.keras import layers
from tensorflow_similarity.layers import MetricEmbedding
from tensorflow_similarity.models import SimilarityModel



text_input = tf.keras.Input(shape=(1,), dtype=tf.string, name='text')
x = vectorize_layer(text_input)

# Next, we add a layer to map those vocab indices into a space of dimensionality
# 'embedding_dim'.
x = layers.Embedding(max_features, embedding_dim,mask_zero=True)(x)
#x = layers.Dropout(0.5)(x)

# Conv1D + global max pooling
x = layers.Conv1D(search_embedding_dim, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.Conv1D(search_embedding_dim, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.GlobalMaxPooling1D()(x)

# We add a vanilla hidden layer:
x = layers.Dense(search_embedding_dim, activation="relu")(x)
#x = layers.Dropout(0.5)(x)
x = layers.Flatten()(x)
x = layers.Dense(search_embedding_dim, activation='relu')(x)
outputs = MetricEmbedding(search_embedding_dim)(x)

# Build a specialized Similarity model
model = SimilarityModel(text_input, outputs)

Train me please:

# Specialized metric loss
from tensorflow_similarity.losses import MultiSimilarityLoss, CircleLoss, PNLoss
from tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping

reduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.2,
                              patience=5, min_lr=0.0001)

early_stopping = EarlyStopping(
    monitor="loss",
    min_delta=0,
    patience=20,
    verbose=1,
    mode="auto",
    restore_best_weights=True,
)

# Train Similarity model using PN loss
model.compile('adam', loss=MultiSimilarityLoss())
model.fit(sampler_train, epochs=500, callbacks=[reduce_lr, early_stopping])

Evaluating Model

We should reindex the train set

# Index embedded   to make them searchable
sx, sy = sampler_train.get_slice(0,X_train.shape[0])
model.index(x=sx, y=sy, data=sx)

Ok let me evaluate our model. This model is a similarity model but we would like to calculate the accuracy.

We defined a simple method to search for similar document on the training set and we assigned the most frequent class to the document. It is just for test!

from tqdm import tqdm
from sklearn.metrics import confusion_matrix
from sklearn.metrics import balanced_accuracy_score

def _eval_accuracy(model, sampler, max_n=10):
    ''' look for similar and keep the label of the best max_n'''

    y_true = []
    y_pred = [] 
    for i in tqdm( range(0,sampler.num_examples) ):
        
        # get the first doc
        qx, qy = sampler.get_slice(i, 1)
        
        # look for best similar
        nns = model.single_lookup(qx[0], k=max_n)

        # 
        y_true_label = int( np.round(qy[0].numpy() ) )
        y_pred_label = int( np.bincount( [nn.label for nn in nns] ).argmax() ) #assigne pred the most frequent class of nearest document FIXME use the most frequente

        y_true.append(y_true_label)
        y_pred.append(y_pred_label)
        
        if i % 10==0:
            print(i, balanced_accuracy_score(y_true, y_pred) )

    return confusion_matrix(y_true, y_pred), balanced_accuracy_score(y_true, y_pred)

Let's call it:

print( _eval_accuracy(model, sampler_test, max_n=30) )

I do not know if make sense to use tensorflow similarity for NLP, indeed we should use CircleLoss, but we preferred PNLoss.

73% accuracy

AGILE for SW Development vs AGILE for Machine Learning

2021-08-11T12:15:00.012+02:00

Agile methodologies are extremely popular for a wide range of application development purposes. Rather than wait months or years for a software project, the Agile method focused on short iterations with a goal of rapidly delivering benefits for the business owner, and then continuously iterating as requirements and needs become more refined [3].

The basic idea of Agile methodology is to continuously iterate the the 6 most important steps at each sprint (SCRUM) or continuously:

Backlog (we consider Business Understanding already accomplished)
Analyse
Define
Design
Test
Deploy

This simple idea mitigates the risk of misunderstanding and reduce complex planning of project management. AGILE allows greater customer satisfaction who sees the product forming slowly. Of course this requires a good mindset:

ability to split large requirements into sub-deliverables,
propensity for refactoring and
acceptance of the PIVOTing.

Machine Learning

Unfortunately in Machine Learning context these concepts are extreme and stressed to the breaking point of the AGILE method [2]. The root-causes of stress are:

Uncertainty of data (missing data, not enough features, not enough data, quality of data, unlabelled data),
SOTA model in research phase and not ready for prod,
Too big expectation of customer.

Furthermore, the ML models are not real products but enablers for software products.

This uncertainty forces us rethinking the Agile process.

Fig. 1: the Agile process for ML

The only AGILE process for ML (At least that I know of) is the CRoss Industry Standard Process for Data Mining (CRISP-DM) [1].

The CRISP-DM highlights that the big difference between the standard AGILE process of the SW and the ML one: dependency on DATA.

However, even the CRISP-DM does not take into account the post production phases : indeed, it is necessary to monitor the performance of the model and fix the model.

Figure.1 shows the differences between Agile Sw and Agile ML. DEV Step 1 is CRISP-DM; POC evaluation Step 2 considers strong interaction with SMEs (Subject Matter Expert) to refine data and model. Indeed a close cooperation with SMEs should be considered and PIVOT should be carried out if necessary. Finally during production (MVP step 3) we need to monitor the model to rebuild model and/or data. So in other words Agile ML is never ending agile methodology.

Agile ML

Typically, (on industrial context not research 😉 ) we can consider the following execution times:

STEP 1 - DEV:

Data Understanding: 1-2 weeks
Data Preparation : 1 week - 2 months

When it is too long (> 1 month) please consider starting with vertical POC with a restricted amount of data

Model Development: 1 week - 1 month
Model Evaluation : 1 week

STEP 2 - POC evaluation:

SMEs face to Face : 1 week
Data Refinement : 1 week
PIVOT decision point : every 2 months

STEP 3 - PROD:

Productisation: 3 months
Model monitoring: 1 day - 1 week

Fig. 2: Agile ML issues

5 Tricks

Although a good process mitigates risks, we are not exempt from issues; Figure 2 highlights the most important issues; here are some tips from experts:

Start with Business Benefit [1]
Split Use Case on smallest user stories [4]
Deliver MVP as soon as possible
Encouraged non-technical stakeholder involvement [4] [5]
Think about tow different profiles ML Specialist (more researcher than sw dev) and ML Engineer (more sw dev) [4]

and finally

"People focusing too much on what Agile is and what is not are bound to make mistakes."

Refs:

https://www.datascience-pm.com/crisp-dm-2/
https://www.infoq.com/articles/machine-learning-agile/
https://www.forbes.com/sites/cognitiveworld/2020/01/19/why-agile-methodologies-miss-the-mark-for-ai--ml-projects/?sh=33188d0121ea
https://www.iunera.com/kraken/big-data-science-strategy/the-agile-approach-in-data-science-explained-by-an-ml-expert/
https://medium.com/qash/how-and-why-to-use-agile-for-machine-learning-384b030e67b6
https://towardsdatascience.com/an-agile-framework-for-ai-projects-6b5a1bb41ce4

IoT Security and Artificial Intelligence for better industrial applications

2021-06-23T10:51:00.002+02:00