This tutorial assumes you have Ubuntu 24.04 LTS server installed along with an NVIDIA GPU. It serves as Part I of an on-premises Agentic AI server buildout series. Part I, is the minimal viable product. It creates a headless and GPU-accelerated AI server with a client-accesible UI, LLM Runner, LLM, and RAG capabilities via manual document uploads.

1. Part I: Minimum Viable Product – Proof-of-concept (open-WebUI + Ollama)
2. Part II: Pilot – Production buildout with 3-concurrent users (N8N + Dify + vLLM + image generation and recognition + Canvas generation + 2-way voice communication)
3. Part III: Enterprise – Production build-out for a medium-sized business with 100 to 500 concurrent users

As we advance through the stages we’ll deploy enterprise tools. These include deployment and cybersecurity tools such as Ansible, VMs, Wireguard and a commercial AI-powered Next Generation Firewall (NGFW).

Phase 1: Update Packages and Nvidia GPU Drivers

# 1. update OS & Kernel
sudo apt update && sudo apt upgrade -y
sudo reboot

# 2. update nvidia drivers for headless, AI workloads
sudo ubuntu-drivers install --gpgpu
sudo reboot

# 3. manually install nvidia utility package
major_ver=$(modinfo -F version nvidia | cut -d'.' -f1)
sudo apt install nvidia-utils-${major_ver}-server 

sudo reboot

Phase 2: Set-up Containerization

In this phase we achieve software isolation, dependency management, and configuration portability. Install Docker & Nvidia Toolkit. We use Docker so that tools like Dify don’t affect system files. Nvidia’s toolkit allows docker to “talk” to your GPU.

# Install Docker using the official convenience script
# curl -fsSL https://get.docker.com -o get-docker.sh
# sudo sh get-docker.sh
# Or install docker engine and docker compose from official ubuntu packages
sudo apt install docker.io -y # https://stackoverflow.com/a/57678382
sudo apt install docker-compose-v2 -y # 

#Download and install Nvidia's Docker container kit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
#  Fix the Architecture Variable
sudo sed -i "s/\$(ARCH)/$(dpkg --print-architecture)/g" /etc/apt/sources.list.d/nvidia-container-toolkit.list


# Enable GPU access for Docker
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker # sudo systemctl enable --now docker

# Add user to docker group (avoid sudo for docker commands)
sudo usermod -aG docker $USER
newgrp docker # "shortcut" that refreshes group memberships for current session so you don't have to log out

Phase 3: Dockerizing the AI Stack

Create a folder called ai-server and create a file named docker-compose.yml.

This is boss-mode. Use this for maximum flexibility:

# Directory Structure
mkdir -p ~/ai-stack/mvp/data/ollama ~/ai-stack/mvp/data/webui
cd ~/ai-stack/mvp && nano docker-compose.yml

Next create and save the following docker file:

# File path: ~/ai-stack/mvp/docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
#  Useful for external access (e.g., for API calls from outside the Docker network).
#  When used internally (e.g., by Open WebUI), not exposing the port is more secure.
#    ports:
#      - "11434:11434"
    pull_policy: always # disable for production
    restart: unless-stopped
    tty: true # for add'l debug info
    volumes:
      - ./ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1  # may also set to 'all' too
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      #- RAG_EMBEDDING_ENGINE=ollama
      #- RAG_EMBEDDING_MODEL=nomic-embed-text:latest # Consider changing this for Part 2 or 3
      #- ENABLE_RAG_HYBRID_SEARCH=True
# If you need super strong security.  
#     - WEBUI_SECRET_KEY=change_me_to_random_string
# bypass login page for a single-user setup, set the WEBUI_AUTH environment variable to False
      - WEBUI_AUTH=False
    volumes:
      - ./webui_data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

Run: docker compose up -d then download Qwen3 with:

docker exec -it ollama ollama pull qwen3:30b-a3b-instruct-2507-q4_K_M

docker exec ollama ollama pull nomic-embed-text

What This Command Does

1. docker exec -it ollama: Runs a command inside the ollama container.-it spawns an interactive session that shows download progress.
2. ollama pull qwen3:30b-q4_K_M: Downloads the Qwen3 30B model with q4_K_M quantization

Quantized Alternatives of the Same LLM: Lower VRAM Usage with More Accuracy

Alternatively you may download Unsloth’s version: sudo docker exec -it ollama ollama pull hf.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:UD-Q4_K_XL.

The disadvantage is that Unsloth does not update its quants for new updates of the same model. For example Quen3-30b-A3-instrcut was updated in September 2025. But, Unsloth still uses the July 2025 version for its quant.

Unsloth quantization algo (UD-Q4_K_XL) results in smaller (17GB vs 18GB) but more accurate model than the standard (e.g., Q4_K_M). See Daniel Han’s reply here.

GLM-4.7-flash will be used in part 2. However, You may download Unsloth’s version with: docker exec -it ollama ollama pull hf.co/unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL.

Model Housekeeping

Docker Commands

1. View downloaded models:

   docker exec -it ollama ollama list

2. List downloaded models:

   docker exec -it ollama ollama list

3. Delete a model:

   docker exec -it ollama ollama rm 

4. check logs: docker compose log

Security Notes

1. Docker security: Running containers as root (default) is a security risk. Consider adding a non-root user in the Dockerfile or using user: “1000:1000” in docker-compose.yml
2. Open WebUI auth: For production, strongly recommend enabling WEBUI_AUTH and setting a WEBUI_SECRET_KEY

Phase 4: MVP of RAG On Uploaded Documents

RAG functionality is built into webUI. No additional action needed for now.

Resources and References

In LLM quantization, the “bits” (e.g., 4-bit, 8-bit) represent the precision used to store model weights, determining memory usage and computational efficiency. 4-bit quantization reduces memory by ~75% compared to 16-bit (roughly 0.5 bytes per parameter vs 2 bytes), enabling smaller hardware to run large models with minimal accuracy loss (<1–2%).

Common options for GGUF are in the table below.

The diagram below clarifies that the Operating System (Kernel) is stored on the physical NAND (SSD), but is managed by a controller inside the SoC.

+-----------------------------------------------------------+          +----------------+
|                   APPLE SILICON (SoC)                     |          |  INTERNAL SSD  |
|                                                           |          |     (NAND)     |
|  +-----------------------+       +---------------------+  |          |                |
|  | Application Processor |       |   Secure Enclave    |  |          | +------------+ |
|  |    (The Main CPU)     |       |      (SEP)          |  |          | | iBoot (S2) | |
|  |                       |       |                     |  |  Secure  | +------------+ |
|  |  +-----------------+  |Mailbox|  +---------------+  |  |  Channel | | XNU Kernel | |
|  |  |  XNU (In RAM)   | <=======>|  |    sepOS      |  | <---------> | +------------+ |
|  |  +-----------------+  |       |  +---------------+  |  |          | | User Data  | |
|  |  |  App Sandbox    |  |       |  | AES Crypto Eng|  |  |          | +------------+ |
|  +-----------------------+       +---------------------+  |          +----------------+
|              ^                              ^             |           Physical NAND
|              |                              |             |         (Encrypted at Rest)
|      +---------------+              +---------------+     |
|      |   Boot ROM    |              |   UID Key     |     |
|      | (Root/Trust)  |              | (Silicon Fuse)|     |
|      +---------------+              +---------------+     |
+-----------------------------------------------------------+

The “UID” Confusion Resolved

The UID (Unique ID) is a physical AES-256 key fused into the SoC during manufacturing. It is the “Master Secret” of the chip.

Why is it “randomly” there? It provides Hardware-Bound Encryption. Because the SSD data is encrypted using a key derived from this specific UID, you cannot desolder the SSD chips and read them on another Mac. The data is cryptographically married to that specific processor.

The Workflow

1. The SEP takes the UID.
2. It combines it with your Passcode.
3. It generates a KEK (Key Encryption Key).
4. This KEK is sent to the AES Engine to decrypt the SSD.

Physical Boot Sequence (The “Hand-off”)

1. Stage 1 (Hardware): The Boot ROM (inside the chip) starts. It knows nothing about the SSD yet except how to talk to the NVMe controller.
2. Stage 2 (Loading): Boot ROM reads iBoot from the SSD. It checks the signature. If valid, it loads iBoot into the SoC’s internal RAM.
3. Stage 3 (Kernel): iBoot now looks for the XNU Kernel on the SSD.
4. Stage 4 (Execution): iBoot verifies the Kernel’s signature, loads it into the system’s main RAM, and the OS begins to run.

Summary of Storage Locations

This content is password protected. To view it please enter your password below:

Password:

Types of machine learning

• Time Series Analysis: When used for forecasting future values based on historical, time-ordered data, it is a type of supervised machine learning. Some of the most effective methods for time series forecasting are machine learning models, including neural networks and gradient boosting frameworks.
• Principal Component Analysis (PCA): This is a key technique in unsupervised machine learning. PCA is a dimensionality reduction method that finds new, uncorrelated variables (principal components) that capture the most variance in a dataset. It is often used to preprocess data for other machine learning algorithms by reducing model complexity and preventing overfitting.
• Regression: This is a fundamental supervised machine learning technique. It predicts a continuous numerical output based on input features and works by estimating the relationship between variables. A regression algorithm is used to train a machine learning model, which learns to make predictions based on labeled data.

Statistical methods that support machine learning

• Hypothesis Testing: This is a statistical method used to evaluate assumptions about a population based on sample data. In machine learning, hypothesis testing can be used to compare different models, evaluate the significance of features, and validate assumptions about data distribution, but it is not a machine learning technique itself.
• Cohort Analysis: This is a statistical and analytical technique for studying the behavior of different groups (cohorts) over time. While it provides insights that can inform machine learning models (e.g., identifying valuable customer segments), it is a form of descriptive or diagnostic analysis, not a machine learning technique.

Quantitative Finance and Numerical Methods

• Used diffusive stochastic processes (including geometric Brownian motion with local volatility) and numerical methods including: Monte Carlo, finite difference methods) to estimate the fair value of equity derivatives
• Diffusive stochastic processes: These are mathematical models used in quantitative finance to simulate the random movement of asset prices over time.
• Geometric Brownian Motion (GBM): A specific type of diffusive stochastic process commonly used in the Black-Scholes model for option pricing.
• Local Volatility: A more advanced model that allows the volatility of an asset to change over time and with respect to the asset’s price level.
• Monte Carlo Simulation: A numerical technique that uses random sampling to model the behavior of financial assets and find approximate solutions to complex problems, like valuing options.
• Finite Difference Methods: Numerical methods for solving differential equations, which are fundamental to pricing derivatives.
• Analyzed risk on a portfolio of options using stochastic volatility models (Local Volatility, Local-Stochastic Volatility)
• Stochastic Volatility Models: These are used to model the fact that the volatility of an asset, like an option, is not constant but changes over time in a random way.

Machine Learning

• Machine learning methods (regularization, clustering algorithms) applied to financial data sets
• Regularization: Techniques like L1 (Lasso) and L2 (Ridge) that are used to prevent overfitting in a machine learning model by penalizing large coefficients.
• Clustering Algorithms: Unsupervised machine learning methods, such as K-means or DBSCAN, used to group similar data points together based on their features.
• Predictive modeling: The process of using statistical and machine learning techniques to forecast future outcomes.
• Classification: A supervised machine learning task for predicting a discrete class label (e.g., spam or not spam) for a given data point.
• Feature Engineering: The process of creating new input features from existing ones to improve the performance of machine learning models.

Statistical Methods

• Applied statistical methods (including Principal Component Analysis (PCA), linear and logistic regression, time series analysis) to financial data sets
• Principal Component Analysis (PCA): A statistical and unsupervised machine learning technique used for dimensionality reduction.
• Linear Regression: A supervised learning method used to model the relationship between a dependent variable and one or more independent variables.
• Logistic Regression: A supervised learning method used for classification problems where the goal is to predict a categorical outcome.
• Time Series Analysis: The analysis of time-stamped data to identify patterns, trends, and seasonality, often for forecasting.
• Used std deviation (Ỽ), variance (Ỽ2), co-variance, correlation (r-value), linear regression, Q-Q Plots, hypothesis tests, p-value, cohort analysis to inform investment decisions from data
• Descriptive Statistics:
  • Standard Deviation, Variance, Co-variance, and Correlation: Measures of data dispersion and the relationship between variables.
• Inferential Statistics:
  • Hypothesis Tests and p-value: Used to make inferences about a population from sample data and determine the statistical significance of results.
  • Q-Q Plots (Quantile-Quantile Plots): Used to check if a dataset follows a particular theoretical distribution, such as a normal distribution.
• Analytical Technique:
  • Cohort Analysis: A method for studying the behavior of a group of people (cohort) over a period of time.

Programming and Data Analysis Tools

• Python (Pandas, NumPy, SciPy, Matplotlib, Seaborn, PyTorch), SQL, R, Excel VBA, Bash
• Python Libraries (Pandas, NumPy, SciPy, etc.): Packages used for data manipulation, scientific computing, visualization, and machine learning.
• SQL: A language used for managing data in relational databases.
• R: A programming language and environment specifically designed for statistical computing and graphics.
• Excel VBA: A programming language for creating macros in Microsoft Excel.
• Bash: A command-line shell used for interacting with an operating system.

Here’s what’s actually being used in production right now:

Agentic AI Frameworks (Orchestration & Multi-Agent Systems)

Tier 1 – Production-Ready (Most Common)

LangChain / LangGraph (Industry Standard)

• Most popular framework for building AI agents
• LangChain = basic chains and agents
• LangGraph = advanced multi-agent workflows with state management

CrewAI (Industry Standard)

• Specialized for multi-agent collaboration
• Agents work together like a “crew” with roles

AutoGen (Microsoft) (Production-Ready)

• Multi-agent conversations
• Good for code generation tasks
• Rising popularity

Haystack (Production-Ready)

• Originally for search, now strong in RAG + agents
• Good for production deployments

Tier 2 – Emerging/Specialized

smolagents (Hugging Face) (Emerging)

• Lightweight agent framework

Google ADK (Agent Development Kit) (Emerging)

• Google’s agent framework

Semantic Kernel (Microsoft) (Emerging)

• Enterprise-focused agent framework
• C# and Python support

LlamaIndex Workflows

• Evolved from RAG-only to include agentic capabilities

RAG Frameworks & Libraries

Core RAG Orchestration

LangChain (Industry Standard)

• Most popular for RAG pipelines
• Document loaders, splitters, retrievers

LlamaIndex (Industry Standard)

• Purpose-built for RAG
• Better for complex document structures

Haystack (Production-Ready)

• Enterprise RAG pipelines
• Good for production deployments

Vector Databases (Storage & Retrieval)

ChromaDB (Industry Standard)

• Most popular embedded vector DB

pgvector (PostgreSQL extension) (Industry Standard)

• Vector search in PostgreSQL

Pinecone (Production-Ready)

• Managed vector DB (cloud)
• Popular in production

Weaviate (Production-Ready)

• Open-source vector DB
• Good for hybrid search

Milvus (Emerging)

• Scalable vector DB
• Good for large deployments

FAISS (Facebook AI Similarity Search) (Production-Ready)

• Vector similarity search library
• Not a full DB, but very fast

Document Processing & Chunking

LangChain Document Loaders (Industry Standard)

• Handles PDFs, Word, HTML, etc.

Unstructured (Production-Ready)

• Advanced document parsing
• Handles complex layouts

PyPDF2 / pypdf (Production-Ready)

• PDF text extraction

python-docx (Emerging)

• Word document processing

BeautifulSoup4 (Production-Ready)

• HTML parsing (for web scraping)

Embedding Models

Embedding models are a specialized type of machine learning algorithm designed to translate complex, unstructured data into numerical vectors. These “vectors” act as a list of numbers that represent the underlying meaning and relationships of the input data

sentence-transformers (Industry Standard)

• Create embeddings locally
• Hugging Face models

OpenAI embeddings (Industry Standard)

• Via API (text-embedding-ada-002, text-embedding-3-small/large)

Cohere embeddings (Production-Ready)

• Via API

Evaluation & Monitoring

LangSmith (Industry Standard)

• LangChain’s evaluation platform

TruLens (Production-Ready)

• RAG evaluation and observability

Ragas (Production-Ready)

• RAG evaluation framework
• Measures faithfulness, relevance, etc.

Phoenix (Arize AI) (Emerging)

• LLM observability

Weights & Biases (Production-Ready)

• ML experiment tracking

Supporting Libraries

Core Python for AI

• NumPy – Array operations
• Pandas – Data manipulation
• Pydantic – Data validation (important for agents!)
• asyncio – Async operations (important for agents)

API & Integration

• requests – HTTP requests
• httpx – Async HTTP
• FastAPI – Building APIs for agents
• Gradio / Streamlit – Quick UIs for demos

Prompt Engineering

• promptify – Prompt templates
• guidance (Microsoft) – Structured LLM outputs