Critical Technology

Converting a Historic Surface Well into a Sauna Splash Pool

2025-11-28T12:05:00.007-03:30

This ambitious DIY project transformed a forgotten surface well into a charming cold plunge pool, creating the perfect complement to a standalone wood-burning sauna. The renovation showcases thoughtful design that respects the well's heritage while adding modern functionality and comfort.

Project Overview

The journey began with a neglected surface well nestled in a rugged, natural landscape. What started as an overgrown hole in the ground evolved into an inviting retreat that combines the invigorating tradition of hot-cold therapy with the beauty of outdoor bathing.

Phase 1: Site Preparation and Excavation

The initial phase required significant excavation work to expose the well's original structure. All vegetation surrounding the opening was carefully removed, and the soil around the perimeter was cleared to create a clean working area. This stage revealed the well's depth and helped determine the scope of structural modifications needed.

Further excavation created space for the concrete platform that would become the pool's foundation. A sturdy ladder was essential during this phase for safe access in and out of the deep workspace and to allow access to widening the well opening.

Phase 2: Structural Foundation

A carefully designed concrete walls were poured to serve multiple purposes: raising the water level to a more comfortable depth, creating a solid base for entry and exit, and incorporating an integrated seating area. The design features concrete block steps and a corner bench, providing both functionality and comfort. The concrete work included proper drainage considerations and smooth rounded finishing to ensure safe, injury-free surfaces.

As the concrete walls cured, the pool's basic structure took shape, complete with the addition of cleaned small rocks to be fill under the concrete slab pool deck.

Phase 3: Enhanced Access Features

The two pictures illustrate the completed concrete structure of the splash pool conversion, specifically focusing on the new integrated deck and steps.

Description of the Structure

The images display the finished concrete basin and the surrounding platform, which was constructed to convert the old surface well into a usable splash pool.

Pool Deck Access to the Deeper Pool

The creation of the concrete pool deck slab surrounding the central water basin is crucial for easing access to the deeper pool. This structure effectively raises the entry point relative to the original deep well opening, creating a wide, stable platform from which users can safely enter the water. The deck provides a manageable step down into the water, rather than requiring a large drop from the exterior edge of the structure.

Three Concrete Stairs Enabling Access

The picture below clearly shows a set of two prominent concrete steps (stairs) built into one corner of the structure. These steps are designed to enable simple access onto the main pool deck. They provide a convenient, integrated way for a person to transition from the surrounding grassy area onto the concrete platform and approach the water's edge.

Ladder Placement and Functionality

The installation was designed to allow for both secure placement and dynamic functionality.

The ladder is anchored to the bottom of the splash pool using a specific pivoting mechanism. This mechanism incorporates eye bolts embedded in a concrete block which is set into the pool floor.

To ensure the ladder could pivot while remaining securely attached, bolts were installed into the bottom of the ladder. These bolts, paired with washers, connect to the embedded eye bolts. The washers facilitate smooth rotation, allowing the ladder to pivot from its base.

This pivoting capability serves two primary purposes:

During pool entry and use: The ladder can be moved out of the way by pivoting it to the side, maximizing open space within the pool basin.
During pool exit: The ladder can be leaned over (pivoted upward) to ease the process of climbing out of the pool.

The completed installation, which includes the wooden ladder, provides easy entry into the splash pool alongside the newly constructed concrete structure and the integrated three concrete steps.

Phase 4: Finishing Touches

The finished installation features a beautiful slatted wooden deck that extends over one corner of the pool, creating a comfortable staging area for entry and exit. The custom-built wooden ladder is angled for easy access, and natural stone placement around the perimeter helps blend the modern construction with the surrounding landscape. The pool now holds crystal-clear water, ready for post-sauna plunges.

Key Features:

Raised Water Level: The concrete platform brings water to an ideal depth for comfortable cold plunging
Integrated Seating: Built-in concrete bench provides a place to acclimate gradually or simply sit and enjoy the water
Safe Access: Multiple entry points including a sturdy wooden ladder and stepped concrete areas
Natural Integration: Stone placement and minimal landscaping disturbance maintain the rustic, natural aesthetic
Wooden Deck: Provides a dry surface for towels and creates a defined transition space between sauna and pool

The Result

This conversion successfully honors the well's historical presence while creating a functional, beautiful addition to the sauna experience. The combination of modern concrete work, traditional woodworking, and natural stone creates a harmonious blend that looks like it's always belonged in this landscape. The splash pool now offers the perfect cool-down after time in the wood-burning sauna, embracing the time-honored Scandinavian tradition of contrast bathing in a uniquely personal setting.

Architectural Integrity Meets Primal Wellness: Engineering a Sauna for Optimal Health and Durability

2025-11-23T19:08:00.009-03:30

As professionals in the building environment, we know that true quality emerges when design decisions are made with both aesthetic beauty and structural resilience in mind. I recently completed a project that perfectly illustrates this principle: a deeply integrated sauna structure designed to withstand extreme conditions while offering profound health benefits.

The Challenge of Placement and Access

Building on a difficult site demands meticulous planning and execution. The location of this sauna required significant manual site preparation. Due to the rugged terrain, moving all necessary building materials was a challenging logistical feat, emphasizing the premium placed on every element chosen for the final structure.

Constructing the Perfect Envelope: Defying the Dew Point

When designing a space defined by intense, fluctuating humidity - like a sauna - the issue of the dew point becomes paramount. Condensation forming within the wall cavities leads directly to rot and mold, compromising both the building's longevity and occupant health.

Our crucial design decision to combat this involved constructing robust, layered wall assemblies that deliberately encourage airflow between the building layers. This controlled ventilation system ensures a perpetually dry building by carrying away moisture vapor before it can condense within the envelope. This preventative measure is fundamental to promoting a durable and structurally sound building for decades. Visible elements of the construction include vertical stud framing and insulation materials integrated into the wall structure.

Performance Through Precision Materials

We selected materials not just for their looks, but for their commitment to performance:

Custom-Milled Rhombus Siding: The exterior features unique custom-milled rhombus siding. This deliberate profile enhances the building's aesthetic appeal while serving as an effective rain screen, supporting the ventilation strategy and rapid drainage.

Triple-Glazed Windows: Windows are often the weakest thermal link. To mitigate heat loss and maintain a high level of energy efficiency, we specified high-performance, triple-glazed windows. This choice minimizes thermal bridging and ensures internal glass surfaces remain warm, further preventing condensation risks inside the heated space.

The custom milled rhombus siding was not only aesthetic, it was also the least expensive.

The Heart of the Ritual: Heat, Cold, and Restoration

The interior is designed around the powerful, transformative experience of the heat and the stones. A moderately sized stove capable of holding a substantial volume of volcanic stones dominates the space, crucial for generating rich, intense löyly (steam). Heating the space is a manual process, requiring tending to the firebox to achieve the desired temperature. Three large spoonfuls of essential oil infused water poured over the hot stones brings the temperature up in 3 - 5 degree bursts.

Adding essential oils to the water before pouring over the hot stones provides additional wellbeing.

The sauna and the associated features are intended to work together as a holistic wellness facility. The structure is integrated with a separate, adjacent cold feature, built by converting the properties legacy surface well. The below-grade concrete structure and ladder is designed for a splash or cold plunge - that is situated a few steps from the sauna building. A deep breath and quickly lowering yourself into the 6'6" pool allows you to emerge into a different dimension.

The Thermal Cycle for Deep Wellness:

The full restorative experience involves cycling between the heat and the cold. Based on traditional practices, the cycle emphasizes a deep heating step followed immediately by a rapid cooling phase.

A suggested cycle involves:

1. Heating: Spending time in the intense heat of the sauna.

2. Cooling: Transitioning directly to the cold plunge or splash pool for up to three minutes.

3. Rest: Resting outside for another 5 to 10 minutes before returning to the sauna.

This cycle is typically repeated three times to maximize circulatory benefits and promote relaxation. Taking the opportunity the sit and relax in the cool air, or by an outdoor fire, and breathe deeply further enhances the experience.

It should be noted that while the existence of the heated sauna and the cold plunge structure is supported by the sources, the specific timing, duration (3 minutes in the pool, 5-10 minutes outside), and repetition count (3 times) for this wellness protocol are details that need to be considered by the user. Know your body and how it responds to heat and cold cycles. Good to consult with your physician before using a sauna and cold pool.

Sunset and views from the sauna are outstanding. And the deck provides enough space for laying about and completing a few yoga postures.

The colour of the wood burning firebox always adds to the ambience and provides enough light to choose a few more pieces of wood organized under the interior bench seats.

The dry summer months also provide an opportunity to maintain the lands around the sauna and splash pool.

This project demonstrates that successful architecture transcends mere shelter; it is about engineering environments that actively support health, durability, and a profound connection to beneficial natural rituals.

Navigating Data Governance in the Age of AI

2025-02-19T22:12:00.001-03:30

The Data Guardian.

Data governance has become crucial in the age of AI, particularly with technologies like Retrieval Augmented Generation (RAG) that combine language models with internal and external knowledge sources. Whether used personally by individuals at home or organizationally for customer service, AI systems' effectiveness depends entirely on the quality and governance of their underlying data. This guide explores five essential elements of data governance for AI systems: data provenance (tracking data origins), data lineage (mapping data journeys), data quality (ensuring accuracy), data security (protecting information), and data access (managing permissions). Understanding and implementing these elements is vital for building trustworthy AI systems that can deliver accurate, unbiased, and compliant results while fostering innovation and protecting sensitive information.

Introduction: Retrieval Augmented Generation (RAG) - Your AI Co-Pilot

Imagine having a personal AI assistant that can instantly answer any question, grounded in reliable information. That's the promise of Retrieval Augmented Generation (RAG). RAG systems combine the power of large language models (LLMs) with the ability to retrieve information from external knowledge sources.

Personal RAG: Think of a student using RAG to research a paper. The AI can access a library of academic articles, textbooks, and credible websites to provide accurate and up-to-date information, tailored to the student's specific query.
Organizational RAG: Now picture a company using RAG to improve customer service. The AI can access internal knowledge bases, product manuals, and FAQs to provide instant and consistent answers to customer inquiries, reducing response times and improving customer satisfaction.

But here's the catch: the effectiveness of RAG, and any AI system, hinges on the quality and governance of the underlying data. Just like a faulty GPS can lead you astray, ungoverned data can lead AI to generate inaccurate, biased, or even harmful outputs. That's where data governance comes in.

Why Data Governance Matters for AI: Personal and Organizational Perspectives

Data governance is not just a set of rules, it's a framework for ensuring that data is accurate, reliable, secure, and used ethically. In the context of AI, data governance is crucial for:

Building Trust: AI systems are only as trustworthy as the data they are trained on.
Mitigating Risk: Poor data quality can lead to flawed AI conclusions, increasing the risk of bad decisions and non-compliance.
Ensuring Compliance: Data governance helps organizations comply with data privacy regulations like GDPR and CCPA.
Driving Innovation: High-quality, well-governed data fuels AI innovation and enables organizations to unlock the full potential of their data assets.

Five Key Elements of Data Governance for AI

Here are five main elements of data governance that are critical for both personal and organizational use of AI:

1. Data Provenance: Tracing the Origin

Definition: Data provenance is the "who, what, when, where, and why" of data. It involves tracking the origins of data, how it has been transformed, and who has accessed it.
Personal Use: Imagine using an AI tool to analyze your personal finances. Data provenance would help you understand where the AI is getting your financial data (e.g., bank accounts, credit cards), how it's being processed, and who has access to it.
Organizational Use: In an organization, data provenance is essential for tracking the source of training data for AI models. This helps ensure that the data is reliable, unbiased, and compliant with regulations. Tools like blockchain can be leveraged for provenance tracking of AI assets. Standards, such as those proposed by the Data & Trust Alliance (D\&TA), aim to surface metadata on source, legal rights, privacy and protection, generation date, data type, generation method, intended use and restrictions and lineage.

2. Data Lineage: Mapping the Data Journey

Definition: Data lineage is the chronological journey of data from its origin to its current state. It provides a complete audit trail of all transformations and processes that the data has undergone.
Personal Use: If you're using an AI-powered fitness tracker, data lineage would show how your activity data is collected, processed (e.g., calculating calories burned), and used to generate personalized recommendations.
Organizational Use: For AI applications, data lineage is crucial for understanding how data quality issues may have been introduced during processing. It also helps in debugging AI models and ensuring that the results are reproducible.

3. Data Quality: Ensuring Accuracy and Reliability

Definition: Data quality refers to the accuracy, completeness, consistency, and timeliness of data. High-quality data is essential for building trustworthy AI systems.
Personal Use: If you're using an AI-powered medical diagnosis tool, you want to be sure that the data it's using (e.g., your medical history, lab results) is accurate and up-to-date.
Organizational Use: Organizations need to implement data quality checks and validation procedures to ensure that AI models are trained on reliable data. This includes monitoring data for bias and implementing mitigation strategies. Characteristics which define data quality are accuracy, completeness, reliability and timeliness.

4. Data Security: Protecting Sensitive Information

Definition: Data security involves implementing measures to protect data from unauthorized access, use, disclosure, disruption, modification, or destruction.
Personal Use: When using AI tools, you need to be confident that your personal data is protected from cyber threats and unauthorized access.
Organizational Use: Data security is paramount for AI applications that handle sensitive data, such as customer information, financial records, or medical data. This includes implementing access controls, encryption, and data loss prevention measures.

5. Data Access: Balancing Openness and Control

Definition: Data access refers to the policies and procedures for granting access to data. It involves balancing the need for open access to data for innovation with the need to protect sensitive information.
Personal Use: You should have control over who has access to your data when using AI applications, and be able to grant or revoke access as needed.
Organizational Use: Organizations need to establish clear data access policies that define who can access what data, for what purpose, and under what conditions. This includes implementing role-based access control and data masking techniques. Semantic models also contribute to data anonymization processes.

Conclusion: Embrace Data Governance for AI Success

AI has the potential to transform every aspect of our lives, but it's not a silver bullet. To harness the power of AI responsibly and effectively, we need to embrace data governance as a core principle. By focusing on data provenance, lineage, quality, security, and access, we can build AI systems that are trustworthy, reliable, and beneficial for all.

Call to Action

For Individuals: Take data governance seriously. Understand where your data comes from, how it's being used, and what your rights are.

For Organizations: Invest in data governance tools and processes. Establish clear policies, train your employees, and foster a data-driven culture.

The future of AI depends on it!

In the coming workshop I will cover these topics and more. I will provide a number of use cases and examples of data governance practices for individuals as they increase their knowledge of integrating AI into their daily practices.

Data Governance for Reliable AI: From Source to Insight

2025-02-14T03:59:00.003-03:30

I am building a workshop to help adult students and professionals make the best use of emerging AI tools for both organizational and personal use. Two themes that will be present in the workshop include, using Retrieval Augmented Generation (RAG) for improving context and being mindful of information privacy within the context of AI.

The overall theme of the workshop is to unlock the potential of AI while ensuring quality and reliability. This workshop explores the critical role of continuous improvement, data governance, data provenance, and data lineage in building trustworthy AI systems. Discover practical strategies for implementing robust data management frameworks to address challenges in data quality, compliance, and model performance, leading to more effective AI solutions.

Key Topics:

Continuous Improvement: Learn why iteratively refining data is crucial for reliable AI outcomes.
Data Governance: Understand the importance of data governance in AI.
Data Provenance & Lineage: Discover how tracking data's origin, journey, and transformations enhances transparency, supports ethical practices, improves decision-making, and reduces hallucinations in AI applications.

AI Approaches: RAG

The workshop will also touch on Retrieval-Augmented Generation (RAG), a technique that fundamentally improves AI systems by enabling them to access and utilize specific, real-time information from organizational documents, databases, and knowledge bases, rather than relying solely on their training data. RAG enhances accuracy and reliability, as demonstrated in applications like healthcare and legal work. Unless an organization builds its own Large Language Model (LLM), everything it does with AI could be considered RAG.

For more detail in moving beyond good prompt engineering you need to consider RAG, please consider this blog post for further insight; https://criticaltechnology.blogspot.com/2024/12/rag-and-agents-how-ai-is-learning-to.html

Tools for Data Governance

The workshop will include a demo of NotebookLM. NotebookLM is designed with robust privacy features that make it particularly relevant for Canadian professionals handling sensitive information. The platform's key privacy feature is that uploaded documents are never used to train its AI models, ensuring data remains private and secure.

Most of the demo's will be in using NotebookLM, the RAG tool built by Google. To better understand NotebookLM and its security position, please consider this recent blog post; https://criticaltechnology.blogspot.com/2025/02/keeping-your-data-private-in-notebooklm.html

PIPEDA and Data Governance

It's important to understand Canada's Personal Information Protection and Electronic Documents Act (PIPEDA). While PIPEDA doesn't explicitly address AI, its technology-neutral principles establish crucial guidelines for handling personal data in AI projects. These include obtaining proper consent, limiting data collection, implementing security measures, maintaining transparency, ensuring data accuracy, and practicing accountability.

For more detail of how AI intersects with PIPEDA enjoy this blog post highlighting seven important impacts; https://criticaltechnology.blogspot.com/2025/02/ai-and-your-personal-project-navigating.html

If you are interested in attending this workshop feel free to sign up. All are welcome. Reserve your spot here: https://lnkd.in/ecKQ-reB

Keeping Your Data Private in NotebookLM: A Canadian Professional's Guide

2025-02-12T00:54:00.000-03:30

As a Canadian professional, you understand the importance of data privacy. Whether you're working with client information, sensitive research, or proprietary business strategies, keeping your data secure is paramount. That's why when exploring new tools like NotebookLM, understanding its privacy features is crucial.

NotebookLM offers a powerful way to interact with your documents, but how does it handle your sensitive information? The good news is that NotebookLM is designed with privacy in mind. Here's a breakdown of what you need to know:

Your Data Stays Yours:

No Training Data: Let's get the biggest concern out of the way first. Your uploaded documents are never used to train NotebookLM's AI models. Think of it this way: your data is for *your* use only, and it doesn't contribute to improving the system for other users. This is a critical distinction and a significant advantage for professionals handling confidential material.
Workspace Account Protection: If you're accessing NotebookLM through a work or school account with a qualifying Workspace edition, you get an extra layer of protection. In this scenario, your uploads, queries, and the model's responses are shielded from human review. This is particularly important for professionals in regulated industries or those dealing with highly sensitive data.

A Note on Personal Accounts and Feedback:

If you're using a personal Google account, the situation is slightly different. If you choose to provide feedback on NotebookLM, human reviewers *might* see your queries, uploads, and the AI's responses. Therefore, it's best practice to avoid submitting anything you wouldn't be comfortable sharing if you're using a personal account. Consider this carefully when deciding how to use the platform.

Key Considerations for Canadian Professionals:

Copyright: As always, respect Canadian copyright laws. Ensure you have the necessary rights to share any content you upload to NotebookLM. This is a fundamental principle regardless of the platform you're using.
Terms of Service: Your use of NotebookLM, whether through a personal or Workspace account, is subject to Google's Terms of Service or the Google Workspace Terms of Service, respectively. Familiarize yourself with these terms to fully understand your rights and responsibilities.

The Bottom Line:

NotebookLM is built with privacy at its core. The platform emphasizes keeping your documents confidential and separate from its AI training processes. For Canadian professionals, this is a vital consideration. By understanding these privacy features and adhering to best practices, you can leverage the power of NotebookLM while maintaining the confidentiality of your valuable data. If you have any further questions or concerns, always refer to Google's official documentation and privacy policy for the most current information.

AI and Your Personal Project: Navigating PIPEDA's Privacy Landscape

2025-02-10T22:39:00.000-03:30

Artificial intelligence is rapidly changing the landscape of what's possible, even in personal projects. Many hobbyists and professionals are exploring the power of AI for everything from creative endeavors to data analysis. But with this power comes responsibility, especially when dealing with personal information. In Canada, the Personal Information Protection and Electronic Documents Act (PIPEDA) sets the ground rules for how we handle such data, and it applies even to your personal AI projects.

PIPEDA doesn't specifically mention "AI," but its core principles are technology-agnostic. Think of it as a set of best practices for responsible data handling, regardless of the tools you use. So, how does this impact your AI tinkering? Let's break it down:

Consent is Key: If your AI project uses any personal information, you generally need consent to collect, use, or disclose it. This is crucial, even if you're not selling anything or sharing the data widely. Think about what data your project requires and how you'll obtain consent.
Stick to the Purpose: You can only use the personal information for the specific purpose you stated when you got consent. Don't collect data for one reason and then use it for something completely different without obtaining new consent. Be clear and upfront about your intentions from the start.
Less is More (Data Minimization): Only collect the personal information you *actually* need for your project. Avoid the temptation to gather extra data "just in case." The less you collect, the less you have to protect.
Protect What You Collect (Safeguards): You're responsible for protecting the personal information you collect with appropriate security measures. This is especially important if you're dealing with sensitive data. Think about encryption, access controls, and secure storage.
Be Transparent: Be open and honest about how you're using personal information in your AI project. People have a right to know how their data is being used, even in seemingly harmless projects. Consider a simple privacy notice or explanation.
Accuracy Matters: If your AI project involves making decisions about individuals, you need to ensure the personal information you're using is accurate and up-to-date. Inaccurate data can lead to unfair or incorrect outcomes.
Accountability is Your Responsibility: Ultimately, you're responsible for complying with PIPEDA, even in a personal project. This means being able to demonstrate how you're protecting personal information and adhering to the principles outlined in the Act.

The Bottom Line:

PIPEDA might seem daunting, but its principles are fundamentally about respect for privacy. By considering these points, you can ensure your AI projects are not only innovative but also responsible. Remember, these are just some key considerations. PIPEDA is a complex piece of legislation. If you have specific questions about how it applies to your project, consulting with a privacy expert or legal professional is always a good idea. Protecting privacy is not just a legal obligation; it's the right thing to do.

Getting Started with AI: NotebookLM

2024-12-27T11:11:00.007-03:30

If you followed along with a previous post on a holiday challenge for learning AI you may now be wondering where too next? Great question, shows you have learned about prompt engineering and are now thinking there has to be more. There is more, a lot more. A good set of skills and understanding of prompt engineering would serve you very well, and you could stop there for a while. Particularly, if you iterate your prompts and increase your literacy in creating prompts. And remember AI can help you improve your prompting.

For many people, I have found that once the intermediate understanding of prompting is achieved the question doesn't seem to go to how do I prompt better. The question seems to go to, can this be automated or can my LLM be more subject specific or can the LLM be restrained personally to my own knowledge. I want the AI to be more specific or give more weight to a narrower or more personal domain of knowledge.

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is like giving an AI system a personalized library to reference while it's talking to you. Instead of only relying on what it learned during training, RAG lets AI search through specific documents or data to find relevant information before generating a response. Think of it like a student who first checks their textbook and notes before answering a question, rather than just going off memory. This helps the AI give more accurate and up-to-date answers based on reliable sources.

There are a few online options to provide you a personal RAG platform. Currently, my two favorites are perplexity.ai and NotebookLM. Both these platforms allow you to upload or reference other resources (text, video, and others) to augment (and focus) your use of AI. Really very amazing at supporting you in creating subject specific AI mentors. I strongly suggest you begin to play with NotebookLM.

Consider using NotebookLM

Set up an account (or use it with your existing google account). https://notebooklm.google.com/
Watch a NotebookLM introductory overview video: https://youtu.be/UG0DP6nVnrc?si=2bGoT7ZMI-VKsU6_
Think about business and personal use cases: https://youtu.be/U3SgtCWsjXg?si=eR_ESarUJTHenPki
Consider the history of NotebookLM development at google: https://youtu.be/sOyFpSW1Vls?si=F9gVrxXrc2vihRnf

If you remain curious about where RAG and automated agents fit into all the near future of AI I published a post last week discussing these two innovations with AI. Twenty twenty-five will be an interesting year.

RAG and Agents: How AI is Learning to Think and Act

2024-12-18T09:25:00.001-03:30

Collaborative RAG and Agents.

In the rapidly evolving landscape of artificial intelligence, two technologies are fundamentally changing how AI systems interact with the world: Retrieval-Augmented Generation (RAG) and AI Agents. While both enhance AI capabilities, they serve distinctly different yet complementary purposes in advancing machine intelligence.

RAG: The Power of Grounded Knowledge

Imagine trying to navigate a foreign city using only your general knowledge of how cities work. You might make educated guesses about where to find the downtown area or how the transit system operates, but you'd likely make many mistakes. This is similar to how traditional Large Language Models (LLMs) operate – they rely on their training data to make informed but potentially inaccurate assumptions.

RAG transforms this paradigm by giving LLMs access to specific, relevant information in real-time. Instead of relying solely on their training data, RAG-enabled systems can pull precise information from your organization's documents, databases, and knowledge bases. This means when you ask a question about your company's Q4 2023 results, the AI isn't generating a plausible-sounding response – it's retrieving and synthesizing actual data from your financial reports.

The impact of RAG on accuracy and reliability cannot be overstated. In healthcare, for instance, RAG-enabled systems can access the latest medical research rather than relying on potentially outdated training data. In legal applications, they can reference specific case law and regulations rather than generating generic legal-sounding language.

Agents: From Knowledge to Action

While RAG revolutionizes how AI systems access information, Agents take things a step further by adding autonomous action to the mix. An AI Agent is more like a capable assistant than a simple question-answering system. It can:

Plan and execute multi-step tasks
Interact with external tools and systems
Maintain context across conversations
Learn from past interactions
Make decisions based on evolving situations

Consider a customer service scenario. A RAG-enabled system might accurately answer questions about your return policy by referencing your documentation. An Agent, however, could actually process the return, check inventory for replacements, schedule a pickup, and update your CRM – all while maintaining a natural conversation with the customer.

The Synergistic Future

The real magic happens when RAG and Agents work together. Imagine an AI system that can not only access your entire corporate knowledge base but also take action based on that information. It could:

Monitor market trends and automatically adjust your digital advertising strategy
Analyze customer feedback across channels and initiate appropriate response workflows
Review legal documents and prepare necessary compliance filings
Manage complex project timelines while adapting to real-time changes

Practical Implications for Businesses

The combination of RAG and Agents represents a significant leap forward in business process automation. Organizations can now build systems that don't just provide information but actually complete complex workflows with minimal human intervention.

However, this power comes with responsibility. As these systems become more capable, it's crucial to implement proper governance structures, ensuring that AI actions align with business objectives and ethical considerations.

Looking Ahead

As both RAG and Agent technologies continue to mature, we're likely to see increasingly sophisticated applications that blur the line between knowledge systems and autonomous actors. The key will be finding the right balance between automation and human oversight, ensuring that these powerful tools enhance rather than replace human decision-making.

The future of AI isn't just about smarter systems – it's about systems that can both understand and act upon that understanding in meaningful ways. RAG and Agents are just the beginning of this transformative journey.

Getting Started with AI: A 15-Hour Learning Journey

2024-12-14T15:25:00.003-03:30

Want to become AI-savvy in just two weeks? Here's a focused learning path that requires only about an hour a day. This guide explores four essential themes that will transform how you interact with AI:

Learning from AI Experts: How to leverage AI podcasts to build your knowledge foundation
The Art of Iteration: Mastering the technique of refining your prompts to get better results
Trust but Verify: Developing critical thinking skills to verify AI-generated content
Smart Summarization: Converting lengthy AI conversations into powerful, reusable prompts

Let's dive into these themes through practical exercises and real-world examples that will help you harness AI effectively in your daily life.

Week 1: Building Your Foundation (7.5 hours)

Deep Dive into AI Through Podcasts (2.5 hours)

Start your journey by listening to carefully selected podcasts during your commute or daily routine. I recommend you choose only one or two for your regular listening pleasure:

"Practical AI" by Changelog - Start with episode #236 on "Prompt Engineering Patterns" which provides concrete examples of effective AI interaction.
"Lex Fridman Podcast" - Episode #362 with Anthropic's CEO is particularly enlightening for understanding AI capabilities.
"Google DeepMind: The Podcast" for cutting-edge AI research and developments.
"The Artificial Intelligence Show" by Marketing AI Institute for practical business applications.

Mastering Prompt Iteration (2 hours)

Spend time practicing with AI chatbots, focusing on refining your prompts. Here's a fun example:

Initial Prompt:

"Write about dogs"

Improved Iteration:

"Write a 300-word guide about choosing the right dog breed for apartment living, including considerations for size, energy level, and noise"

Final Iteration:

"Create a comprehensive guide for apartment dwellers considering a dog. Include:

Top 5 breeds suited for apartment living
Exercise requirements for each breed
Noise levels and training tips
Space considerations
Estimated monthly costs

Format this as a practical guide with clear headings and bullet points"

Summary Iteration:

I often ask the chatbot to provide an improved prompt based upon the contents of the session.

"Please rewrite the prompts within this session into a single well-engineered prompt"

Asking the AI chatbot to rewrite your prompt really helps in deepening your understanding of prompt engineering.

And to make things interesting I sometimes ask the chatbot to rewrite the response with different literacy levels.

"Please rewrite this response for a grade five literacy level"
"Please rewrite this response for a PhD literacy level"

I actually find the response for the grade eight literacy level more interesting than the PhD level.

Using the different AI chatbots (3 hours)

There are many emerging AI chatbots, build some prompts within each. Experiment with different AI chatbots: Play, get curious, ask the AI bot to rewrite your prompt, try the rewrites against all these different chatbots, compare and contrast their responses.

ChatGPT: Excellent for creative writing and coding
Claude: Strong at analysis and detailed explanations
Gemini: Particularly good with multimodal tasks
Perplexity: Specialized in real-time information retrieval and citation

Week 2: Advanced Techniques (7.5 hours)

Verification Strategies (4 hours)

Learn to verify AI outputs effectively with these examples:

When you have Historical Facts:

"You mentioned the Wright brothers' first flight was in 1903. Can you:

Provide specific sources for this date
Break down the key events of that day
Highlight any details you're uncertain about"

When you asked for Technical Advice:

"You've suggested this Python code solution. Can you:

Explain why each line is necessary
Identify potential edge cases
Compare it with alternative approaches"

When you wanted Financial Analysis:

"You've provided a financial forecast for my small business. Can you:

Explain the key assumptions behind your projections
Identify potential economic factors that could impact these numbers
Compare this forecast with industry benchmarks
Highlight any areas where you have limited data or uncertainty
Suggest additional data points that could improve the accuracy of this analysis"

These are three examples of verification for your AI outputs. It is always a good idea to request verification as it reduces the AI hallucinations and increases your knowledge of the topic being discussed.

Work through the sessions from last week and write prompts to verify the information in an AI output. Spend a few hours creating verification prompts, ask the AI to write these for you. Improve upon the verification prompts, iterate.

An AI hallucination occurs when an artificial intelligence generates information that appears plausible but is factually incorrect or nonsensical.

Session Summarization (3.5 hours)

Master the art of creating comprehensive prompts from AI sessions. Here's an example:

Original Conversation:

Human: "How can I improve my public speaking?"
AI: [Provides tips about preparation]
Human: "What about handling nervousness?"
AI: [Shares anxiety management techniques]
Human: "How should I structure my speech?"
AI: [Explains speech organization]

Summarized into Single New Prompt:

"Create a comprehensive public speaking guide for beginners that covers:

Essential preparation steps
Anxiety management techniques
Speech structure and organization
Delivery tips
Common pitfalls to avoid

Include specific examples for each section and actionable steps for implementation"

Practical Exercise Examples

Try these exercises during your learning journey:

1. Content Creation:

Ask AI to write a blog post, then iterate three times, each time making it more specific
Example progression:

"Write about healthy eating"
"Write about healthy eating for busy professionals"
"Create a 7-day meal prep guide for busy professionals who have only 30 minutes for dinner"

2. Problem Solving:

Start with a complex problem like home organization
Break it into smaller tasks
Ask AI to verify the feasibility of each step
Create a final, comprehensive action plan

Reminder: Ask AI to summarize a session and all its progressive steps into a single new prompt. Use this prompt in the different chatbots.

Key Takeaways

After completing this learning path, you'll have:

A solid understanding of current AI capabilities and limitations
Practical experience in prompt engineering
The ability to verify and validate AI outputs
Skills to maintain efficient AI conversations

Remember: Success with AI tools comes from systematic practice and refinement. Start with simple queries and gradually increase complexity as you become more comfortable with the interaction patterns.

Pro Tip: Keep a "prompt journal" documenting your most effective prompts and the situations where they worked best. This will help you develop your own library of reliable AI interaction strategies.

Finding Growth in the Gaps: How Career Breaks Fuel My Tech Journey

2024-12-10T08:15:00.000-03:30

As a technology professional, I've discovered an unexpected rhythm in my career - one that turns the spaces between projects into powerful catalysts for growth. Every successful project completion brings not just a sense of accomplishment, but also a valuable gift: a few months of dedicated learning time. These self-directed sabbaticals, occurring naturally in my three-year career cycles, have become essential periods of exploration and reinvention.

Denis Hassabis and Hannah Fry

My current sabbatical feels particularly significant as I navigate the transformative world of Artificial Intelligence. Building upon my foundation in Machine Learning and data science, I've immersed myself in the AI landscape over the past two months. After exploring numerous AI podcasts, I've found two standout sources that consistently deliver valuable insights: "Google DeepMind: The Podcast" for cutting-edge AI research and developments, and "The Artificial Intelligence Show" by Marketing AI Institute for practical business applications.

This deep dive has also included extensive hands-on experimentation with leading Large Language Models (LLMs). Through countless hours working with ChatGPT, Claude, Gemini, and Perplexity, I've developed a nuanced understanding of each platform's strengths and refined my prompt engineering expertise. This practical experience has been invaluable in understanding the real-world capabilities and limitations of current AI technology.

This intensive learning period has already yielded tangible results. I've developed a comprehensive two-week learning module focused on AI fundamentals and practical applications, designed to help professionals enhance their productivity through AI tools. This resource embodies what I find most rewarding about these career interludes - the ability to synthesize new knowledge and share it with others who are eager to embrace technological advancement.

These deliberate pauses between opportunities aren't just breaks - they're investments in staying ahead of the technology curve. Each sabbatical allows me to emerge stronger, more knowledgeable, and better equipped to tackle the next challenge. In an industry that evolves at lightning speed, these learning periods have proven to be my secret weapon for sustained career growth and innovation.

Thank-you Keyin College

2024-10-29T16:52:00.005-02:30

Another successful three years (well, three years and five months) working in an area I love; technology and adult learning. This was a great three years where I utilized my research and experience in integrating agile technologies and techniques into education. As a faculty we managed the complexity, and risk, of integrating an additional 40 students per semester into the program. And best of all, I taught over 200 students JavaScript, Node.js, SQL, PostgreSQL, Mongodb, GitHub, and how all these fit into FullStack development. It was definitely three years and five months well spent.

Three year successes

My career success seems to occur in three year cycles, I've written about this in the past. It would seem that my time with Keyin College follows the same pattern, I have just finished another three year cycle. Background to this three year theme (with focus on adult learning) can be found here;

Gratitude

I like to share the success. I may have been the leader for a few things at Keyin College, but I couldn't have done it alone. These are the things where I am grateful for the staff at Keyin College;

Steve Taylor for encouraging me to participate in the early program design and for supporting me in becoming program head while we onboarded 60 students a semester.

Christa Mitchell for filling the role of career counselor while I was being onboarded to Keyin. So much to learn so little time. Also, for acting as my proxy when important items needed to be escalated to senior management. I know I can be demanding as an employee, I believe we should all be heard and an organization never knows where their diamond on the beach may be.

Sushanta for being good at want you do and your follow-through is exceptional!

Eric Bailey for helping me understand the overall technical architecture of Keyin's network, infrastructure, and approach to IT. Couldn't have made sense of things without you!

Maurice for being 5 hours older than me and providing the wisdom I needed to understand Keyin College (Or understand the best I could).

Roy Brushett and Will Durocher for those brainstorming sessions four years ago where we covered many important topics that defined the overall FullStack curriculum. Where Agile methodologies fit with Software Engineering and with Pedagogy, our use of O'Reilly for text books (saving students and Keyin considerable $$), the programming languages to be used, where source management (GitHub) fit within our Agile approach, Which board (Kanban or Scrum or both) we would use, etc. So good, great to be involved with the initial design and too see it through to teaching many courses as the whole program stabilized.

All the Faculty for allowing me the privilege of being in your service. We reduced risk and successfully went from 18 students to over 50 in one semester. Job well done. Thanks for your support. It was an honour to be your program head.

And of course, the Students! Honestly, I learn more than I teach. Your questions, your feedback, all our 1 on 1's, the tutorials, and encouraging me to develop workshops on an introduction to and intermediate github. Students are the best! We wouldn't be here without you!

Next Steps

I am being drawn back into the technical realm. Give teaching a break again for a while, focus on shipping software and building a team to exceed expectations. I am good at this so why not lean heavily into a startup, manage / mentor a team, leverage my knowledge of big data, database design and administration, draw upon the family (and personal) history of engineering, or even consider a director or executive position (if someone will have me, I'm so much better in a supporting role).

If you need someone with my background or experience please reach out. Thanks again Keyin College!

Blogging in a time of AI

2024-05-14T20:56:00.006-02:30

I am renewing my frequency in blog posting. This will come after an almost 5 year break from blogging. I am returning because I am back to working on open-source software, educational projects, and the digitization of oceans. And I know I have a learning and success journey to share.

Some Background

I started blogging twenty years back. Yes, 20 years. I was an early adopter and I was involved with technology startups and how the internet was influencing education. At this time I was a big believer that blogging was about content creation. Adding to the collective of the internet by adding meaningful and descriptive content rather than only being a consumer of content. To date, I have published over 500 posts to my assorted blogs. Most of this work was in the first 10 years of my blogging. I essentially posted once a month for the first 10 years. I did have a year where I posted over 100 times. As a summary this is how I posted over the last 20 years.

2004 to 2014 - I published 420 blog posts with good readership. I had a year where I posted more than 100 times. I set this as a goal / experiment to see if I could post twice a week. I did this while holding full-time work, which meant many early mornings and late evenings writing. I learned a whole bunch and my writing skills improved.

The themes for these first ten years was mostly;

technology leadership for startups,
hard-core technology and approaches,
innovative and emerging education,
and the intersection of these three.

2015 to 2024 - I published only 41 blog posts during this 10 years, and nothing in the past three years. Honestly, I was distracted by raising my family of three and doing a whole bunch of life living. Not so focused on work and career advancement.

The themes for the last ten years have mostly been;

integrating with technology community in St. John's NL (I moved),
continue my work on digital badges and micro-credentials,
development of an ocean data startup (still a work in progress),
and working the idea of a reference architecture for the digitization of oceans.

Most exciting of all this is more than 1/2 million views during the 20 years and at some points having over 2500 weekly views. What have I learned from all this blogging? Mostly, that having to write and publish openly to the internet helps the overall community knowledge and it helps me learn more deeply in these chosen subjects.

Next Steps

Again, I will use blogging as my cognitive gymnasium. My subjects themes haven't changed and I will focus upon two main subject areas and continue with updates to this critical technology blog;

Education technology, Heutagogy, and the self-directed learner.

The Happy Heutagogue - https://heutagogue.blogspot.com/

Many things related to, and in support of, the authoring a reference architecture for the digitization of oceans.

The Ocean Set CTO - https://oceanset.blogspot.com/

And my continued musings about technology through my gen X view of the world.

Critical Technology - https://criticaltechnology.blogspot.com/

With all my work and R&D efforts I will learn a bunch of stuff and apply this to the real world through the successful projects I am a part of. I will reflect upon these successes and all that I have learned I will create content that can provide further learning for those around me. And hopefully they will also be entertaining reads.

Collaborating with your AI partner

Blogging has changed for me. There has been a lot of technical and social change since I did most of my blog posting over a decade ago. I had a few focused subjects I was very passionate about, and I wrote about them often. I wrote unincumbered for I would consider myself an early adopter and there was less people publishing to the internet in my chosen subjects. Today there is much more content covering these subjects. And all this content comes in photos, videos, audio, and written articles. Artificial Intelligence is doing a great job of creating and summarizing the content which addresses the complex audience needs and their questions and prompts. Content creation has changed. For a human content creator I believe our work needs to be more intelligent, critical, and creative. Content creators in a time of AI need to do what the AI cannot; daydream, reflect on unrelated subjects, see unlikely connections, be critical, add meaning, create new content that falls between the generated content, fact check and confirm, and add more human intelligence.

How will my blog writing process change? Um, it already has...

I must reflect and draw upon my mastery and do my best to add the new content that AI cannot... AI needs our creativity because it has already parsed the published body of human knowledge. For more insight on my approach, use your favorite large language model chatbot (ChatGPT, Gemini) with the prompt 'limitations of generative AI' followed by the prompt 'How would you suggest a human writer overcome these AI limitations'.

Step by Step my blogging will partner with AI and follow this basic approach;

Capture ideas for new posts, be verbose, be imaginative, think about context
Put these ideas to incomplete blog posts, work ideas for days, for weeks...
Read extensively, add to the understanding of any specific idea
Keep references, cut and paste to the bottom of the related incomplete posts
Prompt AI with phrases from the idea generation
Take blocks of text from written ideas and push them into generative AI, be critical, harvest what you can.
Take the written blog post and ask AI for a rewrite. Change your audience. be critical, harvest what you can.
Try and see, try and write, what AI cannot... add to the body of human knowledge.
Add story telling to improve the overall post
Find pictures to support the writing, format for engagement. Use AI to generate images from passages of text taken from the blog post.
Format, edit, improve, repeat. Be bold... Publish.
Use AI to improve the quality of the writing... Publish again.
Rest, reflect, improve... Publish again.
Yes, I am an advocate to publish before writing is perfect. Publish and then make improvements over the days and weeks that follow. Once the post is considered finished finished... promote it on social media.
Identify what is most important about the post and rewrite for the LinkedIn business audience. Publish to LinkedIn.
Repeat...

Suggested reading: https://www.weforum.org/agenda/2023/02/ai-can-catalyze-and-inhibit-your-creativity-here-is-how/

The Leanstack Way

2021-04-09T12:38:00.000-02:30

The Oceans of Data Lab is honored to be a part of PropelICT's startup accelerator. We had our kickoff meeting a few days ago and the current focus is on learning the Leanstack methodology and using lean canvas to tease out ALL the important details for success. The inline supporting learning modules that are available through leanstack are very helpful.

The Lean Canvas - numbers indicate the order of completion

I feel fortunate that I have been familiar with Agile and Lean approaches for over 20 years. I've got two favorite sayings I use when running software teams, and I like to think I run many aspects of my life with an Agile / Lean mindset.

Ship and ship often (deliver new releases as often and frequently as possible)
Fail and fail often (take risks, innovate, don't apologize, keep moving), success comes from failure.

For me the use of Lean in startups all began with Eric Ries when I watched a YouTube interview of Eric conducted a decade ago. This interview became a part of a 2011 blog post where I describe lean approaches within the Director of Technology role. Since this time I have revisited the works of Eric Ries every few years, he has a lot of useful insights to lean startups. One of my all time favorites in the talk he gave at Google 10 years back.

Google Talk: Eric Ries and the Lean Startup

ODL Newsletter - March 2021

2021-04-08T09:27:00.005-02:30

The Oceans of Data Lab (ODL) monthly newsletter is also finding its footing. It is still going to include monthly updates to the progress we make AND it will start with a few articles of interest within the data labs technology world. I am discovering so many interesting technologies and approaches within the data realm. I'm going to fold my 30 years of data experience into why I believe these are of interest to those working with large amounts of data.

Apache Data Lab

The Apache data lab that comes from the same organization that has brought us so many of the important technologies over the years. And specifically, to think of all the big data technologies they have delivered in recent years... there are just to many to list. What I like most about the data lab is its ability to be deployed to the big three cloud hosting environments. Super smart given the storage and compute requirements for data projects shouldn't be the responsibility of Apache.

DataOps and the DataKitchen

DataOps is a fairly recent concept / term that is about seven years old... and it makes sense that it becomes a discipline in itself as it is not DevOps for Data, it is so much more. The DataKitchen looks to be doing some amazing work in this capacity and have published a good read to help get your head into this important and emerging technology space.

I'm another 3 people into working towards my 100 conversations. It is said that you need to have 100 conversations as you solidify your business / startup idea. So I managed to get another three conversations in. I know this isn't that many, but that's ok as this month was more about setting up technology and thinking about risk, revenue, and the escalator pitch for the startup. I still need to talk with people, and I need peoples help, always. If you know anyone who works with analyzing data or works for a business that has a growing interest in their data, I'd love to talk with them.

What has changed this month?

Over this month my thinking has broadened and become more focused on the needs of organizations and their data. No real pivot, but clarifying what the business will be. The changes fell into three main themes;

A broader interest in helping people with their data. The backstory to my career has always been the information technology around the data. For 30 years I have focused on managing, moving, and building software for the data. This will continue with the data lab. We are still interested in ocean data and a reference architecture for the digitization of oceans, these subjects will become part of the bigger data lab.
It's a Data Lab. It became very clear this month that what I was wanting to do is stand up and run a data lab. I had a great conversation with Graham Truax at Innovation Island and this identified the alignment with my accelerator pitch and the data lab concept. After I re-read my proposal (and subsequent acceptance) to the PropelICT accelerator I confirmed... the startup is focused on creating a data lab with related products and services.
Start with a services focus, rather than product. We need revenue and the data lab is not a small product with a near MVP that can generate revenue. There are a number of MVP's that could bring business value for our customers, but nothing with significant revenue possibility. So our focus needs to be on services where we can leverage the skills and knowledge of the founder and identify projects that align well with the overall vision for Oceans of Data Lab.

It's been a business and technology focused month

This really was a more technology focused month. It was getting all the infrastructure in place to have the lab, fetch some data, and display a basic analytics dashboard. So while setting things up, we weren't that focused on reaching out to potential customers.

Setting up servers
Installing, configuring, and securing analytics software (ELK stack)
Identifying and registering domain names, setting up websites.

What are the risks and assumptions?

We also thought about what are the business risks and what assumptions are we making that could work against our success. We are not going to get into these in detail, writing them down and publishing them helps attract attention and hopefully getting the feedback we need to reduce the risk and prove or disprove the assumptions. We are also focused on what can be a product rather than what is a service.

Assumptions

Companies / Organizations will participate in a publish - subscribe business model for data sets
The data lab concept for preparing data sets for publishing will become accepted by SMB

Risks

MVP doesn't generate enough revenue or provide business value
Primary founder having knowledge, energy, or bandwidth to keep up the pace
Finding skilled employees with deep understanding of data engineering
High cost of cloud based infrastructure

Where is the revenue?

The transactional costs in the publish and subscribe (every data set transaction earns money)

this is definitely my riskiest assumption

SMB pay for services in preparing the data sets for themselves and the marketplace.

does the rise of the data engineer role show a willingness to pay for data preparation

What do we consider our Escalator Pitch?

These are early times and we don't yet have a story to tell. Gak! The escalator pitch is hard, and we really don't know what we are doing when it comes to an escalator pitch.

We help SMB realize new revenue possibilities from their existing data.
We reduce the cost of data preparation for their internal analysis and business intelligence.
We provide the services and technology to help you make sense of all your data.
We make it easy for you to see the value and opportunities based upon your unique business data.

Next Steps:

We need to focus on the customer. We need to find the customers and talk to them.
We need to reduce our risk and prove, or disprove, our assumptions.
We need a technical platform to host an Minimum Viable Product (MVP). I need to identify and prioritize a few MVP's.

If you find the Data Lab an interesting idea or have the need to bring greater value to your existing data, please feel free to contact me. We are building a business and we want to help you bring greater value from your data.

It's Alive! The Elastic Stack as our Data Lab

2021-03-23T20:40:00.003-02:30

So much technical work, so little time! I finished my first three sprints toward standing up the data lab. Standing up infrastructure from scratch so you have clean new compute power is fun, and also a lot of work. Particularly when you include; doing it right, taking no short cuts, and making sure it is secure.

Sprint 0: Setup Ubuntu 20.04 Server with ELK stack.

This was mostly rehydrating virtual server infrastructure I hadn't used in 8.5 years. It needed an upgrade from all perspectives and had a completely new OS. I implemented the ELK stack and made a couple of security changes to lock it all down. I ran a few tests by setting up a couple of websites, getting the JSON confirmation from ElasticSearch, and called up the Kibana dashboard. Oooo... sweet success!

Sprint 1: Vulnerability Assessment. Security changes if required.

This evening I spent some time poking at the overall vulnerability of the server and with the ElasticSearch and Kibana services. I made a few additions and changes for further locking down the services and believe they are as secure as they can be for this first release. Very happy to feel reasonably confident about it's being locked down. Maybe, I'll get lucky and get some free PEN testing. ha.

Sprint 2: Identify and register some well aligned domain names.

I registered the following domain names, even considered buying one... it would have been too expensive. I'll implement the data lab on the oceansofdatalab.com site when it becomes closer to being a minimal viable product (MVP).

oceansofdatalab.com
oceansofdatalab.org
oceansofdatalab.net
oceansofdata.net
sevenseasofdata.com
sevenseasofdata.org

ODE Data Lab has its technology footing

2021-03-18T21:56:00.004-02:30

This month has become more about standing up technology than it has been talking to people about their ocean data needs. That's ok... if you are building a technology company, you need to build technology. Ocean of Data Endeavours (ODE) is about building and utilizing software towards making it super easy to work with data, large amounts of data.

The last 10 days have been about refreshing a server infrastructure I stood up 12 years ago for a number of other projects. What was left was a couple of simple websites, some domain hosting, and all the related mail server infrastructure. All of this needed a complete refresh to be brought up to date;

Rebuild the server infrastructure to have more horsepower. - DONE
Upgrade the Ubuntu OS from 10.04 (Lucid Lynx) to 20.04 (Focal Fossa). - DONE
Rework all the domain aliases to remove dependency on a domain I no longer owned. - DONE
Do some basic security work to the server. Mostly SSH focused. - DONE
Create a new mail server, and do some mailbox maintenance. - DONE
Install Apache2 httpd host. - DONE
Configure Apache2 for a couple of web sites. - DONE
Code some basic HTML to confirm the sites are working. - DONE
Celebrate! http://endeavours.com/

So good to have all this done. The server will provide a strong foundation and is well prepared for the ELK stack and the first load of ocean data. So excited!

An Important difference between DevOps and DataOps

2021-03-16T08:42:00.000-02:30

Where DevOps is automation, technology, and delivery focused; DataOps is more customer focused. I like these descriptions from Wikipedia for DevOps and DataOps;

DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality. https://en.wikipedia.org/wiki/DevOps

DataOps is an automated, process-oriented methodology, used by analytic and data teams, to improve the quality and reduce the cycle time of data analytics. While DataOps began as a set of best practices, it has now matured to become a new and independent approach to data analytics. DataOps applies to the entire data lifecycle from data preparation to reporting, and recognizes the interconnected nature of the data analytics team and information technology operations. https://en.wikipedia.org/wiki/DataOps

The similarities between these two are many, particularly from a process and automation perspective. I see DevOps really focused on delivering quality software, and DataOps focused on delivering visualized data analytics to the customer.

Customer focused DataOps assists with Agility

Having a customer focused data analytics team fits well with an Agile approach. The data analytics team needs very involved customer analysts (or product owners). The customer analyst identifies the KPI's, models, or intelligences that need to be fulfilled. These become part of the backlog, and as new sprints are defined they become focused on the item(s) of analytic. A sprint can be built around a few analytics, then iterate around the items for a DataOps sprint;

Where is the data? How do we get at it?
How do we best move it? How often? What are the security or privacy issues?
What needs to be cleansed or transformed? Is the data at the correct granularity?
Do we already have any related data to improve the intelligence? Is this a new build or do we use / alter an existing pipeline?
What models or analytics do we apply?
How do we best visualize the data?

Not to say that DevOps can't fit well within Agile approaches, it can.... the backlog is more technically focused and fits into the sprint more from a continuous perspective than a customer perspective. (What DevOps features go into a sprint are often negotiated with the product owner). The focus of DataOps is in shipping features that fulfill a visualized analytic or more... The focus of DevOps is in CD / CI...

https://medium.com/data-ops/dataops-is-not-just-devops-for-data-6e03083157b7

This approach worked well for us when working on a Business Intelligence project and our nine week sprints usually focused around 4 to 9 KPI's. The organization was in aerospace, they had many legacy data sources with new data sources coming online. As with many organizations, they were in a state of improvement and transformation. Fitting new cubes, representing KPI's, into sprints allowed us to show progress and success. The biggest challenge wasn't in the technical or delivery side of getting the data to the customer. The challenge came in developing a data team where every team member understood the process end-to-end and the efforts required during each step of the DataOps pipeline. Acquiring, cleansing, and transforming data takes as much effort and understanding as visualizing the data for the customer.

For Contract Database Administrator

2021-03-13T11:34:00.007-03:30

Do you require contracted database administration? Medium to small organizations using database technology to store corporate data definitely need database administration to care and feed for there database technologies. This care and feeding includes, and is not limited to;

Install and maintain database servers.
Optimize database security.
Build and maintain ETL pipelines.
Performance tuning of databases.
Storage optimization for databases.
Implement DataOps for up-to-date business analytics and its need for continuous data.
Install, upgrade, and manage database applications.
Create automation, and schedule, repeating database tasks.
Ensure recoverability of database systems.

If you require any, or all, of these database administration task we can help. With over 30 years of database experience complimented with a technology degree in database management we can work remotely to keep your databases healthy and reduce business risk. Part-time or full-time, reasonable rates.

Data Challenge Panel Session: The Power of data

2021-03-12T16:39:00.005-03:30

I listened in on this presentation about the power of data. In particular, having Susan Hunt from Canada's Ocean Supercluster as one of the presenters. All the presenters had very valuable insights. It was an hour well spent!

The Session: The Power of data

Description: Come learn with us. Our panel guests are engaged in transformational data products and projects. Together, we’ll learn what opportunities they are creating when using the newest technology to exploit the power of data. And we’ll talk about careers. The opportunities are limitless.

Moderator: Cathy Simpson | Chief Executive Officer, TechImpact Panelists: Susan Hunt | Chief Technical Officer, Canada’s Ocean Supercluster Justin Kamerman | Chief Product Officer, Instnt Inc. Jason Lee | Partner, MNP Technology Solutions

Items for my follow-up:

A number of subjects sparked my interest from the presenters discussions. I believe these are the three that need follow-up from the current ODE perspective:

Building Models for Data, or transforming Data for Models
What is the business case for the datalab / data workbench?
What is the current state of DataOps?

Peter, we need to integrate data more easily.

2021-03-12T11:25:00.009-03:30

I liked this article on Fundamental truths when it comes to innovators. I do think Elon Musk and Jeff Bezos have figured out how to be crazy successful in business. I do wish they were more philanthropic with the monies they have accumulated from their success. I digress...

I like the idea of a fundamental truth as a foundation for your business that doesn't change through time. And when I think about my commitment to build a data services business, I have now started to think about what would be the fundamental truths?

We need to access and integrate data more easily.
We need better ways to visualize, communicate, and understand, the data.
We would like to reduce the compute and storage costs of data.

These are what I came up with through my review of my initial thinking of fundamental truths. I know there will be a few more and these three will be edited as my idea grows and gets greater footing.

Building the Data Lab Technology Stack

2021-03-08T13:37:00.003-03:30

The idea of building a data lab is emerging from my ocean data conversations and how to best utilize my knowledge and skillset within this opportunity. In my mind, the service offering would be twofold;

Data Engineering / Software Development consulting and services with focus on ocean data. We will do the heavy lifting of extracting, cleansing, transforming, and loading your data. And then we will help with analysis and visualizing the data. We are comfortable working in both the open source and Microsoft technology stacks.
Standing up (and data loading) the technology stack for the data lab. You are going to need to host all this compute power and storage somewhere. It could be on-premise. Most likely, it will be in the cloud. We can help with this also. We could build it in Azure, using the Microsoft technology stack. Or we could build it using an Open Source stack on top of Linux in any of the hosted environments of Azure, AWS, or Rackspace.

https://stock.adobe.com/

How do you build a low priced, large compute, technology stack to support data engineering efforts, implement a data lab, and showcase these new services capabilities. The low price is the key factor given the current startup state of this ocean data endeavour. Particularly, when you think of the cost of compute for processing and storing large amounts of data. I believe the the best way forward is as follows;

Use open source where you can. Fortunately, many of the infrastructures, tools, frameworks, and programming languages for the data lab are open source.
Automate the build so it can be built and torn down with ease. This would eliminate the need for the stack to always be running.
Store the data at it's source, if possible. Fetch, and load, the data when you automatically rebuild the stack. Keep in mind this limits the amount of big data you can store locally, and loading large amounts of data can cumulatively take days. Be mindful of this.

Note: this stack is to showcase the services capabilities. A full data lab would also need the ability to both persist and fetch data. It's going to take some time to build the data lab!

The Data Lab Technology Stack

The deployment of this technology stack will use open source wherever possible running on a Linux (Ubuntu) Server hosted at Rackspace. The rational for these decisions are;

Little to No licensing costs
Strong familiarity with Rackspace as hosting company
Existing domain name (endeavours.com) hosted with Rackspace
Extensive experience with Ubuntu Linux in a hosted environment
Familiarity with deploying data intensive solutions using the ELK stack
Experience programming in Python

Note: The deployment of this technology stack will happen in phases, where each phase will complete with some basic tests to ensure the stack behaves as desired.

Phase 0

Phase 0 will be a basic ELK stack running on an Ubuntu Linux server hosted at Rackspace with access via the endeavours.com web domain. The use case for where the data comes from, how we transform it, the analysis, and visualization is still to be determined. This use case will be used for testing this first iteration of the newly stood up data lab. Exciting times!

Phase 1

During phase 1 we will add the Python programming language to the technology stack and use it for two purposes;

Apply a model to the data using Python.
Present the processed data to a web page for display.

Phase 2

During phase 2 we will add Kafka as an infrastructure resource, identify some additional data sources, and pre-process the data before it gets loaded into ElasticSearch.

Phase 3 and beyond

Investigate the Apache Data lab stack, add Spark to our lab, add a data workbench...

ODE Newsletter - February 2021

2021-03-05T14:56:00.055-03:30

I'm 7 people into working towards my 100 conversations. It is said that you need to have 100 conversations as you solidify your business / startup idea. So this is where I am, seven conversation in. If you know anyone who works with ocean data or works for a business that has an interest in oceans, I'd love to talk with them.

Given the time restraints of being deep into a large data / database migration project, I consider February has been a good month for conversations. It provided me a good view into the horizon of ocean data. I followed the conversations that were presented to me without me directing the focus. For this is the first month, and I have yet to gain clarity of the gaps of where I need more information. This makes sense given I am at the beginning and don't know what I don't know. Now that it is the end of February I have identified the need to talk with customers of ocean data. This could become a focus for March. The conversations for February unfolded in the following order, with the following summaries and highlights;

PropelICT (https://www.propelict.com/)

I reached out to a past co-worker in a leadership position within PropelICT. PropelICT is an Atlantic Canada e-accelerator for tech startups. The conversation was very encouraging and initiated my application to their April cohort. Looking forward to their support in the coming months (and years).

Highlights:

The idea of 100 conversations.
My first suggested conversation contact.
Being a candidate for their e-accelerator.

eOceans (https://www.eoceans.co/)

I spoke with one of the principals of eOceans. Time very well spent, Thank-you! So many details to be digested from this conversation. This organization clearly understands ocean data and where it intersects with social media! A bulleted list seems the best to call out the highlights;

There are many open standards and organizations working in this space. The data standards seem to be "standardizing" and there are many organizations working toward bringing the data standards together. More open organizations are contributing than the closed proprietary types. CIOOS is the standout for Canada. EU and US are much further down the standards and open data path than Canada.
Both ends [(data storage and end-points (IoT)] of the data collection are well serviced with lots of business and startup activity. It's the middle were the greater opportunity exists. It's with the data integration with consideration for all the standards and granularity. "It would be nice to dust off a 10 year old data set and be able to easily use it".
Working with ocean data initiatives is very project based and finding the revenue sources / the business model for an open reference architecture for the digitization of oceans could prove difficult.

Highlights:

Many open organizations already working in the ocean data space.
The business side of what you are exploring (reference architecture) may be difficult, so much work is project based and gov't funded. A reference architecture seems like an NGO or consortium kind of thing.
Middle ground of software and data integration could be a big need given my skillset.

Mentorship

Super fortunate to reconnect with an older friend who has loads of experience; small devices, programming, data, startups to a favorable exit, machine learning, etc... many skills that align well with what I am doing. And on top of all this, I really enjoy the meandering conversations we share!

The one area where there is a strong overlap towards my ocean data focus and the mentors previous experience with the integration of data. And yes he confirmed, integrating data from different devices to a common standard is a lot of work for creating a single view into a broad data realm.

Highlight: He agreed to provide me mentorship within this endeavour. So great!

New Brunswick Ocean Strategy: Our Opportunities in the Blue Economy

This was an excellent online conference put together by the Ocean Supercluster. What I did most was listen, and a good thing too... I have so much to learn. I really liked the breakout sessions where there was more individual participation. Some names, and acronyms are becoming more familiar too me.

Highlight: A small list of contacts I could reach out to. All good!

TechNL (https://www.technl.ca/)

I spoke with one of the leaders in TechNL and we talked about what I am wanting to do with data, in particular, ocean data. The conversation pointed towards two relevant contacts;

Petroleum Research of Newfoundland Labrador (PRNL) and how they are always focused on innovation within the Oil and Gas.
OceansAdvance which is focused on the Newfoundland and Labrador Ocean Technology sector.

Highlight: That if I am going to be successful in this endeavour I am going to need partners. The time required for setting up an organization isn't the best place for me to be focusing my time at this stage of the startup. And given the nature of this startup needing to work in the open, the partnership route may be the best way to go...

Canadian Integrated Ocean Observing System (https://cioos.ca/)

So fortunate to have the attention of two CIOOS employees! They were so gracious a provided a broad and deep amount of information regarding the state of ocean data. Super helpful! CIOOS clearly knows the data. The best way to summarize my conversation is by including the important questions and there answers;

With ocean data where is the greatest pain?
Resources as in financial and skills / knowledge.
At the more general project level; governance and the people who know how to organize and stewardship data through its lifecycle. This is more a reference to the industry in general... it's a project issue. And having the ability to integrate with a project that happened years ago...
Do open data standards have an influence?
Absolutely! There are many references to open data. Most of what we deal with are open.
How easily integrated are the existing data sets?
It’s getting better. It can be difficult to get an older data set and want to integrate it. These older sets often lack the granularity or metadata that makes it easier to ingest. There is a definite need here at a project level. Developing an expertise here could become a strong business.
Most initiatives within this space are project based. Which makes it difficult for longer initiatives that have some data sustainability. Rarely are there long term funding initiatives.

Highlights:

So many acronyms, references and URLs. The CIOOS folks provided me many references all pointing in the right direction. Reference to some of the ISO standards.
The need for better stewardship of data so as data ages it still has usefulness.

Pisces Research Project Management (https://piscesrpm.com/)

Another fortunate conversation with a person deep into ocean data and with the added bonus of being very technical. This was a contact I harvested from the New Brunswick Ocean Strategy Conference. There are may topics I could summarize from this outstanding conversation, much of the information confirmed things I discovered from the previous conversations described above. This is good!

I did pitch my idea about mooring buoys as a fixed points of data collection, and having these buoys like the personalized weather stations that have become so popular. This employee loved the idea.

The exciting part of this conversation was the discussion of the technical stack used within the open data within the oceans sector. It was good to add this to the knowledge I had of the proprietary technical stack used when I was managing the software engineering dept. at Provincial Aerospace.

What is the most common tech stack for Ocean Data?
This person has extensive experience working with Government Organizations and Academics. From what they have seen the most common, and emerging, technology stack includes;

Python
Assorted data storage approaches. Often NOT an RDBMS.
QGIS is common.

These are the tools he finds most effective and common. Using QGIS pushes you into the geo representation of data. Much ocean data requires different kinds of models, more 3d, more oceans… not necessarily geographic, etc.
The ability to prove models with real data is the biggest need from a technical perspective. This is why python has such good traction. It is easy for non-programmers and also rich enough for programmers. A good language for data, and useful across the technical skills working with data.
NetCDF is the most common data-store. Also CSV and proprietary data storage. Remember data people are mostly not programmers or overly technical.
Also take a look at CKAN (https://ckan.org/)
What are people looking for from a technical perspective?

Proving models with real data.
Integrating data

Highlights:

A deep discussion about the technical stack. The preferred programming languages, data storage, integration approaches, and technical issues.
Confirmation that integrating data and proving models is an area of software development opportunity.

Lessons Learned

A reference architecture for the digitization of oceans is not enough to hang a startup or business upon at this time! Where I do believe it is still a good idea that will form through time. There is so much work already going on for a common open architecture that another doesn't need to be started. I truly believe a reference architecture will emerge, it is a; when it will happen, not if it will happen.
There is a big need for technical and software development skills and knowledge in the data engineering space of ocean data. I believe the opportunity exists for a software development / data engineering consulting firm with the specialty of ocean data.
The idea of an anchored (or fixed) buoy for ocean data collection is very compelling too me. Kind of like the personal weather station but as a fixed mooring buoy. Anyone who has a mooring buoy could replace it with the data buoy, and have real-time data about the conditions at the buoy in preparation for mooring.

Next Steps

March will be the month of broadening my reach. I need to talk with a broader section of people working in the oceans space. I need to find potential customers for the processing and software development in, and around, ocean data.
I need to start building software tools for the processing of ocean data. I need a reference technology stack showcasing our abilities to work with data.
I need to start developing an elevator pitch for the ocean data software consulting firm. I need customers and revenue to get the real feedback to focus the business mission.

The Beginning of Ocean Data Endeavours (ODE)

2021-02-21T17:21:00.006-03:30

Thirty-nine months ago I started on a deep dive into developing a reference architecture for the digitization of oceans. The idea of developing this reference architecture was initiated by the Canadian Government awarding Atlantic Canada with the Ocean Super Cluster initiative and all my recent work with leading the software engineering group at Provincial Aerospace. My writing and research into ocean data took me to the point where I needed to deepen my understanding of a number of subjects, and I needed this deeper understanding before I could continue the writing and research (even though you could consider deepening understanding as research). I needed to have an intermediate understanding of what had come before and the current state of things with a reference architecture for the digitization of oceans. In particular, I needed to work directly with ocean data and the standards that influence its structure.

Over the last three years I have been lucky to work with Triware Technologies Inc., and together we have found projects that align with this need to deepen my understanding of all things digital and all things ocean. My recent project successes include;

OCIO Digital by Design - I was fortunate to be awarded the opportunity to be the data architect for the initial phase in digitizing the NL governments citizen facing portal. I remained on the project for the first 12 months through to the portals launch. Being on the design team to create the data tier and integrate with legacy data was a great achievement. And I deeply enjoyed using a scrum / jira approach with a multi-vendor, multi-disciplined team. We achieved a lot in a short period of time.

Lessons Learned - Agile, Scrum and Jira can scale well to a government organization with multiple scrum teams working toward an integrated solution.

Ocean Sector Search - We needed a way to index the Canadian Ocean Sector. So we built a search engine seeded by as many oceans related URLs as our analysts could gather. The technical architecture of this ocean specific search engine can be found in this previous post.

Lessons Learned - With reasonable technical effort Nutch can be configured and seeded to crawl a specific industry sector (in this case Canada Oceans Sector). The Nutch crawl harvested a significant number of pages (> 32000) that were then loaded into the ElasticSearch (ELK) stack while relevancy scoring each page along the way.

NLCHI - My work with the Newfoundland and Labrador Centre for Health Information (NLCHI) was a quick engagement to focus their requirements backlog into a few manageable sprints. I was super fortunate to help get an important project underway and gain insights into the concept of a customer focused data workbench for a specific subject domain.

Lessons Learned - The idea of a personal data workbench is very compelling when you consider the number of data sets already available in the oceans sector. And if we could fold in open and proprietary data sets, while honoring security and privacy we may be onto something...

Nalcor Energy Database Consolidation - So many databases, so little time. One of my favorite enterprise type projects is when the project pays for itself, over time, by the savings created by the projects downstream accomplishments. Not revenue generation, but operational expense savings. I believe one of the best KPI's for IT is not new systems implemented, but old systems retired.

Lessons Learned - an amazing amount of data can be moved with the correct use of tools, well built and managed ETL (pipelines), and a mindset of automation.

Argo Floats - 2018

NEXT STEPS

Over the last month I have revisited how to best develop my intermediate understanding of oceans data. After a number of conversations, with experts of oceans data, I believe my next steps are twofold; I need to focus on the existing standards for oceans data and I need to write some code to integrate some open oceans data sets.

I need to find opportunities to work directly with oceans data. If you are in the oceans sector, in any way, and you have the need of a very experienced data engineer, then I would love to help with your project. If you know of an oceans data project in needs of a data engineer, please forward on my credentials. Thanks to everyone for reading this far. And thank-you Triware for your ongoing career support!

Digital by Design, Agility and Data Architecture

2020-03-19T16:33:00.010-02:30

For 12 months, starting the summer of 2018, I was very fortunate to fill the data architect role for the Government of Newfoundland and Labrador's digital by design citizen facing web portal. An amazing team was brought together and we accomplished an amazing amount of work given the complexity of the environment we were all working. Kudos to the leadership team for seeding the ground and pulling together a diverse and effective group of people.

Being the oldest team member, with 35 years as a technology professional, I noticed a number of items and approaches that I consider the highlights of the project. I call out my 35 years experience because I know success doesn't always happen in a large group of people (with a team larger than 35). A group of strangers doesn't always come together when tasked to ship software on schedule and on budget. The cool part of this project is that the highlights were both technical and project management. In a nutshell, we came together using a scrum model of project management (hosted within JIRA) and architected a microservices technology stack using predominantly Microsoft technologies. The user experience design was exemplary and the software approach stayed aligned with the best of agile practices. We also used a scrum of scrums approach to manage the three distinct scrum teams.

What made this first year of a new project so effective?

The Agile Practices
The team was encouraged to use Agile approaches to successfully ship software. Thankfully, the commitment came from the most senior level and agile workshops were used to align the teams understanding and approach to agile. I consider these three agile practices what kept us all well aligned;

We rigorously stayed with 3 week sprints. This was facilitated by the scrum of scrums group and kept us all focused on shipping working software.
We embraced jira and stayed true to moving cards. It took a few sprints, as a whole we ended up having all the team members updating and moving cards. This, combined with morning standups, kept the team transparency high and important issues in the open.
We always had demo days and retrospectives. This went a long way to keeping us focused and successful. All team members were encouraged to attend the other scrums demo days, this built excitement and kept us focused and moving.

Software Engineering Discipline
Developing software is as much art as it is science. Our team included many accomplished software engineers and this helped us implement features quickly and completely. Kudos are deserved by many on this project team, in particular, one of our technical leads (this is you, Phil) was hellbent and lead through example with two attributes of software engineering that are super important and sometimes missed;

We refactored always, no excuses. As a group we were always learning, as implementing features is a relentless teacher. Improving upon our code base through refactoring kept the quality improving, and the bugs low. Even from a data architecture perspective, at the beginning of each sprint we refactored the data tier with the required data changes from the previous sprint. Data tiers often have different heart beats that the middle and user tiers as they are dependent on the legacy systems, which often have legacy heart beats. This is a blog post in itself...
Automated testing. We automated whenever we could, we aspired to have automated tests with coverage to all our code. We got close by using frameworks and having a test first mind set. And don't underestimate how effective existing testing frameworks can be applied to the data tier.

Architecture was collaborative
All architects were encouraged to contribute and discuss, we were always white-boarding and soliciting feedback. This kept the architecture strong and well understood throughout the team. And because we had a shared understanding of architecture the refactoring was reduced. All good...

Ocean Sector Specific Search

2019-11-26T17:31:00.000-03:30

I've recently finished building an industry specific search engine. The primary use case is to drive international and domestic business traffic to the Canadian websites doing business within the oceans technology and innovation sectors.

From a technology architecture perspective we built a search engine for the Canadian oceans super cluster initiative where all components run, and are based, upon Canadian assets hosted in Canada. We seeded the search engine using the URLs for all the organizations identified as participants within this economic sector. The indexing process analysed each URL and followed all links up to two hops deep. All the identified URLs were scored using a web graph and the top pages were indexed.

The architecture decisions

The NELK stack became our back-end infrastructure.

A number of important architecture decisions were made early on as the project was detailed. Mostly decisions were made to support the technologies that the small team was already familiar. And if the team wasn't familiar, we chose technologies that had the most industry support and local resources in our personal networks so we could help out if we needed. We ended up having Nutch feeding the ELK stack using Wordpress for the UX. In the project it became known as the NELK stack.

Nutch - for web crawling and first round of web page extraction and cleanse.
ElasticSearch (ES) - as the search engine / data manager
Logstash - as the data transform and load.
Kibana - as the administration / developer console

Crawling the web with Nutch
We ended up using Nutch to crawl the internet for ocean sector specific web pages. We also needed to integrate with ElasticPress so the broader ecosystem search included the contents of our websites Wordpress database. To do all this we ended up using Nutch 1.15 for it integrated best across our technology stack. We used the Nutch recommended approach seeding, ingesting, fetching, and duplicate removing as we prepared the data for export to ElasticSearch. Due to versioning issues we exported the Nutch database to CSV before importing the data. For the first load of data our use of Nutch created the following page loading metrics;

seeded with 2612 domain names
removed 709 duplicate or in error domain names
identified 86872 candidate webpages
fetched the 29323 most relevant web pages (based upon web graph algorithms)
indexed 29270 pages into ElasticSearch

Loading data with Logstash, inspecting the results in Kibana
We used Logstash to bring the Nutch exported CSV data into ElasticSearch. Coding up the logstash job was fairly easy, the most important aspect was choosing the correct logstash filter. It was better to use the dissect filter rather than the csv filter. More on this in a later post. In the end, I was amazed with how quickly Logstash loaded, and ElasticSearch indexed, all the data.

Once all the data was loaded into ElasticSearch I used Kibana to confirm data was correctly loaded into the data repository. Kibana has a very intuitive interface and creating filters and running queries to confirm the successful loading of data was straight forward. I look forward to using Kibana to manage the repository and create meaningful dashboards.

Integrating ElasticSearch with WordPress PhP

Integrating with Wordpress

Once we had the back-end built and loaded with industry specific web pages we still needed to find the correct tool-set to provide a query interface within a Wordpress template and to integrate with the organizations identified in the Wordpress database. We wanted the ecosystem search to include both what we had indexed from the internet and the organizations listed in our directories database. The solution ended up using two solutions;

The ElasticSearch (ES) PhP library which provides a mature (and easy to use) set of features to build your own interface into ES using PhP.
ElasticPress which allows automated ElasticSearch integration with a wordpress database.

The Wordpress / PhP tools for integrating with ES are very effective. ElasticPress has automation that keeps it up to date as changes are made within the Wordpress database. The ES PhP library has a robust set of features that makes for easy coding and kept search performance very high. Even with large query results the ability to traverse the result set forward and back was easily handles with features available in the PhP library.

In conclusion, using Nutch with the ELK stack provides for a very strong search engine that integrates easily with Wordpress on the front-end. The learning curve for this approach was not overwhelming and whenever challenges presented themselves the online groups help us resolve issues within days.

Special thanks to the team put together by Triware Technologies. Without all the other technical people, analysts, business people, data entry, project managers, Oceans Advance, ACOA, Ocean Super Cluster, ElasticSearch support, Azure support, and those clearing the way... none of this would have been possible. Thank-you!