Data is everywhere, but?

Still Unsure Which SCD Type to Use?

noreply@blogger.com (Dinesh Asanka) — Thu, 30 Jul 2026 18:03:48 +0000

One of the biggest challenges in designing a data warehouse or modern analytics platform is deciding how to preserve historical data.

Most data professionals are familiar with Slowly Changing Dimensions (SCDs) Type 1, Type 2, Type 3, Type 4, and Type 6 but choosing which type to use for a particular business scenario is often the real challenge.

Should you overwrite the existing value?
Should you maintain every historical version?
Should you store history in a separate table?
Or is a hybrid approach the better option?

The decision depends on your business requirements, reporting needs, query performance, and the frequency of data changes.

I've created the following decision diagram to help identify the most suitable SCD type based on common real-world use cases. I hope it serves as a quick reference for data engineers, BI developers, and data warehouse architects.

Which SCD type do you use most frequently in your projects?

#DataEngineering #DataWarehouse #DataModeling #DimensionalModeling #SlowlyChangingDimensions #SCD #ETL #BusinessIntelligence #Analytics #DataArchitecture #SQLServer #MicrosoftSQLServer #DataEngineeringCommunity #DataAnalytics

Geometric Mean in Power BI

noreply@blogger.com (Dinesh Asanka) — Mon, 06 Jul 2026 05:36:10 +0000

In my latest article, I walk through how to calculate the Geometric Mean in Power BI, including practical implementation steps and when you should use it instead of the traditional average.

Key takeaways:
🔹 Understand the difference between Arithmetic Mean and Geometric Mean.
🔹 Learn why growth and compounded values require a different approach.
🔹 Implement Geometric Mean calculations in Power BI with practical examples.
🔹 Improve the accuracy of your analytical dashboards and business reports.
Whether you're a Power BI developer, data analyst, or BI professional, understanding the right statistical measure can significantly improve decision-making.
📖 Read the full article here:
https://lnkd.in/gYwrP4za

I'd love to hear how you've handled compounded metrics or growth-rate calculations in your Power BI projects.
PowerBI BusinessIntelligence DataAnalytics DAX DataVisualization Statistics DataScience Analytics BI PowerBICommunity

Calculating the Harmonic Mean in Power BI

noreply@blogger.com (Dinesh Asanka) — Tue, 16 Jun 2026 04:02:49 +0000

I am pleased to share my latest article published on SQLServerCentral: “Calculating the Harmonic Mean in Power BI.”

In data analysis, we often use the arithmetic mean to summarize multiple values. However, the arithmetic mean may not always give the most meaningful result, especially when the values have large differences. This is where the harmonic mean becomes useful.

The harmonic mean is particularly important when we need a balanced measure across multiple values. For example, in ranking scenarios or model evaluation, we may not want one high value to hide the weakness of another value. A common example is the use of precision and recall in classification models, where the harmonic mean helps provide a more balanced evaluation.

In this article, I explain the concept of the harmonic mean using simple examples and then demonstrate how it can be calculated in Power BI. Since Power BI does not provide a direct built-in option for calculating the harmonic mean, I show a practical method using Power Query.

The article walks through the process step by step:

First, the data is loaded into Power BI. Then, an inverse value is calculated for each observation. After that, the data is grouped based on the required category, and the count and sum of inverse values are calculated. Finally, the harmonic mean is obtained by dividing the count by the sum of inverse values.

This approach is useful for analysts, BI developers, students, and anyone who wants to apply more meaningful aggregation techniques in Power BI reports.

I hope this article will be helpful for those who work with ranking data, evaluation metrics, and analytical reporting in Power BI.

Read the full article here:
Calculating the Harmonic Mean in Power BI – SQLServerCentral

#PowerBI #PowerQuery #DataAnalytics #BusinessIntelligence #SQLServerCentral #DataScience #Analytics #DAX #BIReporting

🚀 New Publication: Understanding Type 6 Slowly Changing Dimensions (SCD) in SQL Server

noreply@blogger.com (Dinesh Asanka) — Tue, 09 Jun 2026 13:14:41 +0000

Data warehouses must balance two important requirements:

✅ Maintaining historical data
✅ Providing access to the latest / earlier information
One powerful technique for achieving both is the Type 6 Slowly Changing Dimension (Hybrid SCD), which combines the strengths of Type 1, Type 2, and Type 3 approaches. This enables organizations to track history while simultaneously supporting current-state reporting.
In my article published on MSSQLTips, I demonstrate:
🔹 The concept of Type 6 SCDs
🔹 Practical SQL Server implementation examples
🔹 How to maintain both historical and current attribute values
🔹 Benefits and challenges of hybrid dimensional modeling
🔹 Real-world data warehousing applications
If you work with SQL Server, Business Intelligence, Data Warehousing, ETL, or Analytics, this article may help you better manage changing dimensional data while preserving historical accuracy.
📖 Read the full article here:
Type 6 SCD Example in SQL Server

I would be interested to hear how your organization handles slowly changing dimensions and historical data management.

Can Role-Based Learning Transform How We Teach Requirement Elicitation?

noreply@blogger.com (Dinesh Asanka) — Sat, 28 Feb 2026 04:39:00 +0000

In 2010, I learned a powerful teaching lesson — not from a conference, journal article, or workshop — but from my daughter’s first day at school.

Her very first classroom activity was simple: “sellam gedara.”
Children were assigned roles — mother, father, daughter — and asked to interact within those roles.

It looked like play.

But it wasn’t just play.

It was structured role-based learning.

That moment stayed with me.

Years later, I found myself asking:

If role-play is powerful enough to shape social learning in children,
why aren’t we using it more intentionally in higher education?

The Challenge: Teaching Requirement Elicitation

I teach Data Warehousing, and one of the most difficult topics for students is Requirement Elicitation.

On paper, it looks straightforward:

Identify stakeholders
Gather requirements
Document them
Validate

But in reality?

It’s messy.

It involves:

Conflicting priorities
Miscommunication
Negotiation
Role tension
Ambiguity

And none of that can be fully understood through slides alone.

So I decided to experiment.

The Experiment: Role-Based Learning in Action

I redesigned the session around structured role simulation.

The Setup

40 students
Divided into 8 groups
Each group received a different business scenario
Roles assigned within each group:
- Client
- Business Analyst (BA)
- Project Manager (PM)
- Developer (DEV)
- Quality Assurance (QA)

The PM was responsible for leadership and coordinating meetings.

But here’s where it became interesting.

The Twist: Dual-Level Interaction

This was not just a simple role-play.

Phase 1: Expert Meetings

All students holding the same role met together.
All PMs discussed leadership strategy.
All BAs discussed elicitation techniques.
All Developers discussed feasibility concerns.

They aligned their professional perspective first.

Phase 2: Team Elicitation

Students returned to their project groups and conducted requirement meetings.

After an hour, they regrouped again by role to reflect:

What challenges are emerging?
What conflicts are surfacing?
What adjustments are needed?

This cycle repeated multiple times.

The Deliverables

After three intensive hours, each group submitted:

Requirement Document
Meeting Minutes
Delivery Plan
Timeline

More importantly, they experienced:

Stakeholder tension
Negotiation pressure
Time constraints
Communication breakdowns

In short they experienced reality.

A Deliberate Decision: Stepping Back

I made a conscious decision not to intervene directly during the session.

Why?

Because when lecturers stand at the center, students often perform for the lecturer.

I wanted them to perform for the problem.

Four junior staff members observed at different levels, but the learning space belonged to the students.

Was It Perfect?

No.

Was it powerful?

Absolutely.

An anonymous survey at the end revealed:

38 out of 40 students expressed strong positive feedback
Observers noted higher engagement
Students demonstrated improved negotiation and ownership
Discussions became deeper and more authentic

Most importantly — the classroom energy changed.

Students were no longer passive recipients of theory.
They were professionals inside a simulated project environment.

What I Learned

Role-Based Learning:

Increases cognitive engagement
Strengthens procedural understanding
Improves collaboration skills
Makes abstract concepts tangible

It also introduces productive discomfort and that discomfort creates growth.

Why This Matters

Today’s graduates do not work in isolated roles.
They operate within cross-functional teams.

If we expect industry-ready professionals,
our classrooms must simulate industry complexity.

Slides cannot replicate stakeholder conflict.

Role interaction can.

A Small Pedagogical Risk

This was not a perfect model.
It was an experiment.

But innovation in teaching rarely begins with certainty.
It begins with thoughtful risk.

And sometimes, the most meaningful transformations in education start with something as simple as children playing “sellam gedara.”

I am currently exploring more structured research on role-based peer interaction and measurable learning outcomes in computing education.

If you are experimenting with active learning strategies in higher education, I would love to hear your experiences.

Let’s reimagine the classroom together.

Vs of Data

noreply@blogger.com (Dinesh Asanka) — Sat, 10 Jan 2026 08:03:00 +0000

Flipped Classroom: Reclaiming the Lecture Hall Through Practice and Experience

noreply@blogger.com (Dinesh Asanka) — Thu, 08 Jan 2026 03:15:00 +0000

For many years, my classroom experience—both as a student and as an educator—reflected a familiar pattern in higher education. Students entered lecture halls, listened quietly, took notes, and left once the lecture ended. Questions were rare, discussion was limited, and learning largely remained a passive activity.

Over time, this began to feel deeply unsatisfying. Students were present, but not necessarily engaged. The lecture hall functioned more as a space for information delivery than for learning.

Why I Decided to Change the Model

I wanted to break away from this pattern, but the question was how. The opportunity came when I was introduced to the concept of the flipped classroom. What initially seemed like a simple reversal of teaching order turned out to be a fundamental shift in how students interact with content, peers, and the classroom itself.

Instead of using classroom time to explain theories, the flipped model emphasizes preparation before class and active engagement during class. I decided to adopt this approach in a senior-level computing module, fully aware that it would require significant effort and experimentation.

How the Flipped Classroom Was Implemented

Each week, I created and shared short video lectures and carefully selected readings through the learning management system. Producing these materials was not easy—it was time-consuming and intellectually demanding—but it allowed students to engage with the content at their own pace.

Students were expected to review these resources before coming to class. Classroom time was then redesigned around problem-solving, discussion, and collaborative activities rather than traditional lecturing.

What Changed in Student Behavior

One of the most striking outcomes was a visible change in student behavior.

Students who previously arrived late began coming to class early—sometimes 20 to 30 minutes in advance—to review materials and prepare for discussions. Instead of passively waiting for the lecture to begin, they came with questions, opinions, and ideas.

Classes typically started with a short formative question based on the pre-class materials. Students evaluated each other’s responses and then moved into group discussions. The quality of interaction improved noticeably. Students spoke more, questioned assumptions, and explained concepts to one another.

Importantly, the use of tools such as AI-based assistants, online documentation, and external resources was not restricted. These tools became part of the learning process rather than something to be avoided. The focus shifted from “getting the right answer” to understanding why an answer works.

The Role of the Lecturer

My role in the classroom changed significantly. Instead of spending most of the session explaining content, I spent time listening, asking probing questions, and guiding discussions. This made student thinking more visible and allowed misconceptions to be addressed immediately.

The classroom became a shared intellectual space rather than a one-directional delivery channel.

Challenges and Reflections

This approach is not without challenges. Preparing high-quality pre-class materials requires sustained effort, and not all students adapt immediately to increased responsibility for their own learning. The physical layout of traditional classrooms—fixed rows and rigid seating—also limits interaction and collaboration.

However, even within these constraints, the benefits were clear. Engagement increased, attendance became more meaningful, and learning felt deeper and more authentic.

An Ongoing Process

I do not view the flipped classroom as a final or perfect solution. It is an evolving practice that must be continuously refined based on student feedback, learning outcomes, and contextual constraints.

Each semester provides new insights into what works, what does not, and what can be improved. That iterative process is, in many ways, the most valuable part of the experience.

Concluding Thoughts

The flipped classroom transformed not only how my students learned, but also how I approached teaching. It challenged long-standing assumptions about lectures, authority, and classroom control.

If our goal in higher education is to cultivate critical thinking, collaboration, and independent learners, then reclaiming classroom time for active learning is essential. The flipped classroom offers one practical pathway toward that goal.

I strongly believe that the most meaningful improvements in teaching come from reflecting on practice, observing student behavior, and being willing to change—even when it is uncomfortable.

Vs of Data

noreply@blogger.com (Dinesh Asanka) — Wed, 07 Jan 2026 17:52:00 +0000

🚀 New Video Release! | Cricket Analytics Meets Machine Learning 🏏🤖

noreply@blogger.com (Dinesh Asanka) — Tue, 16 Dec 2025 16:44:00 +0000

As a lifelong cricket lover and a researcher in data analytics, I'm excited to share my latest work that digs deep into a long-held belief in T20 cricket — Do extra deliveries (wides & no-balls) really give a major advantage to the batting side?

Using data from 269 T20 matches across international & franchise formats (both men’s and women’s cricket), I applied multiple machine learning models and feature selection techniques to uncover the truth.
📊 Key Insight:
Despite what many experts say, the outcomes of extra deliveries are highly unpredictable — behaving more like random events than strategic advantages.
🔍 What the study reveals:
73% of extra deliveries have low or no positive impact
9 ML models could not reliably predict outcomes (<40% accuracy)
Only a few parameters (match stage, batsman strike rate, bowler runs conceded) showed weak influence
Expert intuition doesn’t always match data-driven reality
🎥 I have prepared a YouTube cover/thumbnail summarizing the findings visually.
Full video: https://lnkd.in/gYib62QY
Full Paper : https://lnkd.in/gWrcX9Xg
Always happy to connect with others who love cricket, analytics, or both! 🏏📈
Cricket T20Cricket MachineLearning SportsAnalytics DataScience Research BigData CricketLover AzureML OrangeDataMining LinkedInResearch DataDrivenInsights

noreply@blogger.com (Dinesh Asanka) — Mon, 15 Dec 2025 17:56:00 +0000

Learn how to conduct a PRISMA-compliant Systematic Literature Review (SLR) using Zotero in this video https://lnkd.in/gBPGdqrC. Whether you're a researcher, academic, postgraduate student, or practitioner, this guide will walk you through the full PRISMA workflow — from database searching to exporting the PRISMA flow diagram.

In this video, you will learn how to:

✅ Set up your Zotero library for systematic reviews

✅ Import and organise research articles from multiple databases

✅ Remove duplicates using Zotero’s advanced tools

✅ Apply PRISMA-based screening (title, abstract, and full text)

✅ Use tags, notes, and metadata to track inclusion/exclusion

✅ Export data for

PRISMA documentation and reporting

✅ Generate and prepare a PRISMA 2020 flowchart

This tutorial is ideal for:

– Researchers conducting literature reviews

– Academics teaching research methods

– Anyone wanting a clean and efficient PRISMA workflow

Have you captured all research papers for your study?

noreply@blogger.com (Dinesh Asanka) — Sun, 30 Jun 2024 16:29:00 +0000

Started writing articles with LinkedIn platform as it has many readers.

Have you captured all research papers for your study? | LinkedIn.

Promtology: Prompt Patterns for Prompts

noreply@blogger.com (Dinesh Asanka) — Sun, 02 Jun 2024 17:20:00 +0000

Generated from Co-Pilot

Design patterns are like blueprints for solving common problems in software development. They aren't code snippets, but rather established templates or strategies that describe how to structure your code to make it more maintainable, flexible, and efficient.

Similar to design pattern in software development, there are pattens for other domains as well. Due to the increase usage of GenAI tools such as ChatGPT, Gemini or Co-Pilot, it is important identify the prompt patterns. By reading few research papers, following list of prompt patterns were identified.

Prompt Patterns contains 17 prompt patterns that are categorized into six categories.

Data Warehousing Best Practices

noreply@blogger.com (Dinesh Asanka) — Fri, 05 Jan 2024 18:43:00 +0000

Continuing on the Data warehouse article series, this is the next article, Data Warehousing Best Practices for SQL Server at www.mssqltips.com. I have updated the list of articles as well at Data is everywhere, but?: All Data Warehouse Related Articles.

All Data Warehouse Related Articles

noreply@blogger.com (Dinesh Asanka) — Sat, 09 Dec 2023 16:08:00 +0000

Writing is my passion. Writing has opened me many avenues over the years. Thought of combining all the data warehouse related article into a one post in different areas in data warehousing.

DESIGN

What is a Data Warehouse? (mssqltips.com)

Things you should avoid when designing a Data Warehouse (sqlshack.com )

Infrastructure Planning for a SQL Server Data Warehouse (mssqltips.com )

Why Surrogate Keys are Needed for a SQL Server Data Warehouse (mssqltips.com )

Create an Extended Date Dimension for a SQL Server Data Warehouse (mssqltips.com )

SQL Server Temporal Tables Overview (mssqltips.com)

Data Warehousing Best Practices for SQL Server (mssqltips.com)

SSIS

• Testing Type 2 Slowly Changing Dimensions in a Data Warehouse (sqlshack.com)

• Implementing Slowly Changing Dimensions (SCDs) in Data Warehouses (sqlshack.com)

• Incremental Data Extraction for ETL using Database Snapshots (sqlshack.com )

• Use Replication to improve the ETL process in SQL Server (sqlshack.com )

• Using the SSIS Script Component as a Data Source (sqlshack.com )

• Fuzzy Lookup Transformations in SSIS (sqlshack.com )

• SSIS Conditional Split overview (sqlshack.com )

• Loading Historical Data into a SQL Server Data Warehouse (mssqltips.com )

• Retry SSIS Control Flow Tasks (mssqltips.com )

• SSIS CDC Tasks for Incremental Data Loading (mssqltips.com)

SSAS

• Multi-language support for SSAS (sqlshack.com )

• Enhancing Data Analytics with SSAS Dimension Hierarchies (sqlshack.com )

• Improve readability with SSAS Perspectives (sqlshack.com )

• SSAS Database Management (sqlshack.com )

• OLAP Cubes in SQL Server (sqlshack.com )

• SSAS Hardware Configuration Recommendations (mssqltips.com )

• Create KPI in a SSAS Cube (mssqltips.com )

• Monitoring SSAS with Extended Events (mssqltips.com)

SSRS

• Exporting SSRS reports to multiple worksheets in Excel (sqlshack.com )

• Enhancing Customer Experiences with Subscriptions in SSRS (sqlshack.com )

• Alternate Row Colors in SSRS (sqlshack.com)

Cloud

• Migrate On-Premises SQL Server Business Intelligence Solution to Azure (mssqltips.com)

Other

• Dynamic Data Masking in SQL Server (sqlshack.com )

• Data Disaster Recovery with Log Shipping (sqlshack.com )

• Using the SQL Server Service Broker for Asynchronous Processing (sqlshack.com )

• SQL Server auditing with Server and Database audit specifications (sqlshack.com )

• Archiving SQL Server data using Partitions - SQL Shack

• Script to Create and Update Missing SQL Server Columnstore Indexes (mssqltips.com )

• SQL Server Clustered Index Behavior Explained via Execution Plans (mssqltips.com )

• SQL Server Maintenance Plan Index Rebuild and Reorganize Tasks (mssqltips.com )

• SQL Server Resource Governor Configuration with T-SQL and SSMS (mssqltips.com)

Dataset : T20 Outcome of Extra Delivery & Free Hit

noreply@blogger.com (Dinesh Asanka) — Fri, 01 Sep 2023 18:01:00 +0000

Cricket is a fascinating game as it provides much entertainment for spectators worldwide. With the instruction of the T20 format, most countries except New Zealand and Zimbabwe other countries have franchise cricket. Franchise cricket has spanned the global mark of T20 cricket.

With cricket, there are common beliefs such as extra delivery that would be delivered once a bowler sends a wide or no ball, a free hit will have a very high impact, etc.

The following dataset is collected to verify these hypotheses. Data collection is not yet completed as well are you need to wait for results for a couple of months.

T20 OutComeofthe ExtraDelivery FreeHit | Kaggle

Violin Plot - Compare Data

noreply@blogger.com (Dinesh Asanka) — Wed, 23 Aug 2023 11:32:00 +0000

Contrasting and comparing are part of data analysis to make vital decisions. A violin chart integrated with Box Plot is one of the charts that can be used to compare data. Let us see how we can utilize the Orange data mining tool to compare data using the Violin chart.

Let us assume that the following is the dataset that we need to compare.

In this dataset, there are five variables to compare. If you simply use a Violin chart without any preprocessing, you cannot compare charts in one chart. Therefore, there are a few preprocessing tasks to be done.

In the above preprocessing tasks, separate processing tasks are used for each variable. Select Columns, Edit Domain, and Feature Constructor are used. You can download the Orange workflow at ODM/Violine-Blog.ows at main · dineshasanka/ODM (github.com)

Finally, all data flows are connected and then used to Violin Plot, as shown below.

Now you can see from the above chart, it is easily can be compared. In the above example, X1 has higher values.

How Data Science Project Works - From the Koobiyo Teledrama

noreply@blogger.com (Dinesh Asanka) — Mon, 21 Aug 2023 16:49:00 +0000

About five years ago, the Koobiyo teledrama was very popular due to the uncharacteristic nature of the teledrama. It was a political teledrama, one reason the teledrama became popular. However, this post is not to discuss the political side of the tele drama but to discuss the data science side of it.

Jehan, the main character of this teledrama, has the idea of building a tool to predict the future problems of people. He uses decision tree architecture to achieve this and explains his idea to his companion, Priyantha as shown in the below video. Watch from the 540th second.

So his idea is very clear. By using data like Gender, Marital Status, Professional Qualification, Parents' details, Friends' details, etc.

After building the predictive software, he presents his case to his good friend Hiruni and his developer too joins the discussion in the following video. Watch from the 40th second.

After observing the work done by Jehan, Hiruni was impressed, but she had a very vital question to ask. "Why do you think people will buy this". Jehan was a little perplexed by the question, and he stressed that this is the software that everyone will have at their fingertips. Obviously, people know their problems they need solutions, not to suffer from their own problems.

This is something important to us as data scientists. We are very good at technical details, but we need to understand the solution part of the tool. Remember, people need a solution to their problem not a technical tool.

Then Hiruni provide a solution to Jehan and asked him to provide this to a human resource person to predict the future problems of its own employees.

Then the solution can be viewed in the 21st second of the following video.

In the live demo, Jehan proves that it is important to select the best people for the organization from this software. It is clear how we should think from the business point of view rather than focusing on technical aspect mainly. As a data scientist, we need to look at the solution side of it.

Article : Learn about Data Warehousing for Analytical Needs

noreply@blogger.com (Dinesh Asanka) — Tue, 15 Aug 2023 09:35:00 +0000

I started another article series at www.mssqltips.com on data warehousing. Even though there are many tools to build data warehouses, I still find there is a lack of conceptual knowledge in the community. So trying to cover that gap. So, read this What is a Data Warehouse? article and send your comments.

Who is a Data Scientist? A Dragon a Pegasus or a Unicorn?

noreply@blogger.com (Dinesh Asanka) — Sun, 30 Jul 2023 18:09:00 +0000

Unicorns, Pegasus, or dragons are mythical creatures that are created by combining the strong attributes of different animals. In Sri Lankan culture, we can see different dragons in temples. It is believed that these dragons are to protect various people and places.

These dragons have seven parts from different people, as expressed in the following poem.

As per the song, the dragon is a combination of

· trunk of an elephant

· legs of a lion

· ears of a pig

· teeth of a crocodile

· eyes of a monkey

· body of a fish

· wings of a bird

The combination of these most strengthened parts will make the dragon a strong animal to achieve his required tasks.

Isn't that the same with a data scientist? Don't they need different strengths, as shown below?

Does that mean data scientists are mythical? Read the full article here E-Magazine – Magazine that bridges Management and IT

Clustering as Pre-Processing Technique for Classification

noreply@blogger.com (Dinesh Asanka) — Sat, 29 Jul 2023 10:47:00 +0000

Clustering is often used to identify natural groups in a dataset. Since the clustering technique does not depend on any independent variable, the clustering technique is said to be unsupervised learning. The classification technique is supervised as it models data for a target or dependent variable. This post describes clustering as a pre-processing task for classification. This post has used the Orange Data Mining tool to demonstrate the above scenario.

Following is the complete orange data mining workflow, and this is available in the Git Hub as well.

The attrition sample dataset is used to demonstrate the above scenario. This dataset contains 18 variables with 1470 instances. Then the Jaccard distance is used, and it is essential to remember that you may have to choose a different distance technique in order to improve the results.

Then hierarchical clustering is used for three clusters as too many clustering will have smaller clusters and may cause overfitting. Following is the cluster distribution for the three clusters.

After the clustering is completed, the dataset is selected for each cluster and unnecessary columns (Clsuter and Selected) are removed. Then for each cluster, different classification techniques were executed. This scenario uses SVM, Logistic Regression, Neural Network, Random Forest, AdaBoost, Naive Bayes, Tree, and kNN classification techniques.

When classification techniques were executed for the entire dataset, without clustering, the best classification technique was Logistic Regression with a Classification accuracy of 87%.

Following are the classification accuracies and other evaluation parameters for each cluster once it is evaluated with all the above said eight different classification techniques.

The above table shows that C1 and C2 clusters have higher accuracy than the full dataset. However, the third cluster performance is not as good as the overall dataset classification.

This technique is better when there are large volumes of data if not clustering will reduce the data which can lead to model overfitting.

Is GPT-4 a Job Destroyer?

noreply@blogger.com (Dinesh Asanka) — Wed, 05 Apr 2023 16:22:00 +0000

With rise of GPT-4, many jobs will be on the firing line. For example, with the expectation of 10% letter writing will be done with the use of GPT-4, job of secaratries are in danger. What about the software developers.You you can upload a image and get the source code from the GPT-4!!!

Wouldn't that be the same fear we had when the computers were introduced. Being in the Software field in early 1990s in Sri Lanka, I still remember how much pressure we had from many employees when we try to install new software applications. During most of the product demostration, we used to conclude our sessions by addressing that fear. We said that new software system will not destroy any jobs but instead it will create new jobs. Isn't that true? If I look back the those industries back, yes some jobs have gone for good, but there are new jobs.

Let us look at Cricket. During 1980's there were only a couple of camera for TV coverage. With the improvement in technology, now there are many cameras including spider, stump, drone etc. With these technologies, viewship has increased and more revenue avenues are being created. Further, cricket has become more glamoours and into a new world. With that new jobs are created and the good example is the third umpire.

Same will be inline for GPT. Let us wait and see in time to come.

2nd Evaluation of Oxford Mathematical Model during the Knockouts

noreply@blogger.com (Dinesh Asanka) — Mon, 12 Dec 2022 12:15:00 +0000

We are into the business end of the sporting event, FIFA 2022. As we have been doing for last couple of weeks, let us evaluate the Oxfor MAthematical Model. Even though the much anticipated semi-finals are yet to be kicked off, we can do the full evaluation. It is because, even now we know that the oxford predication for the final wont be a correct, as Brazil and belgium are out of the competition.

Let us analysis the stage wise accuracy rates.

Problem with this accuracy calculation is that the error is propagated. Since Oxford predicted that the Belgium will feature in the final in which they were out of the competition from the first round, that is reflected at every stage.

Would you accept this model with just over 50% of accuracy. That means it is just over the ad-hoc probability prediction. As we have been insisting, predicting the results outcome of a sport event, is not yet successful, it least for the moment.

Evaluation of Oxford Mathematical Model after First Round

noreply@blogger.com (Dinesh Asanka) — Sat, 03 Dec 2022 06:35:00 +0000

We have been discussing the FIFA results prediction that was developed by Oxford University. Since the first round is completed, we can now do a model evaluation with the actual data vs. predicted data. Oxford mathematical model has two components, predicting the teams for the next round and the teams' position at each group.

As we saw in FIFA 2022, there were many surprises. While Japan, Australia, Morocco, and South Korea advanced to the next round, we saw Belgium (The number 2 ranked team), four times champions, and Germany out of the world cup in the first round.

Following is the final 16s.

Out of the Oxford predictions for the next round, 9 out of 16 are correct, which is 65% of accuracy. Let us look at the group-wise rankings. Out of the 32 positions, only 11 were correct, meaning the accuracy is less than 35%.

When the group-wise analysis was made. All Group F position predictions were wrong, while there are no groups that all positions are correct.

When the position analysis was made, there were a few notable observations.

We can see a higher accuracy for positions 1 and 4. That means it is easy to predict the best and poorest teams. This is not again a hard part. However, their model could be better when predicting the number 2 and 3 positions. In fact, the number 3 position prediction accuracy is 9%.

Sport is all about fractions, inches, and moments. That is where the prediction of results is not difficult but impossible. If it is possible, it is not a sport.

Prediction of FIFA results went wrong, AGAIN!

noreply@blogger.com (Dinesh Asanka) — Wed, 23 Nov 2022 15:58:00 +0000

No matter what level of mathematical models you can build, still predicting the result of a sport is extremely difficult, if not impossible. FIFA 2022 started with a lot of predictions, and we are just into the 4th day and 12th match, yet we have watched the two greatest upsets in the FIFA world cup.

The first, I would say it is the greatest upset of all time in which Saudi Arabia won against Messi's Argentina.

At the end of the first half, Argentina was leading 1-0, but in the next half, the oil nation scored two goals quickly. After the Saudi's first goal, the Argentinian giants were in disarray. In the end, most of the analytics said that crowd support for Saudi Arabia was one of the reasons for their surprising win.

The second match was between Japan and four times world cup winners (1954, 1974, 1990, 2014) Germany.

Germany was showing their class during the first half and scored the first goal. Germany has never lost a world cup match after 1954 after scoring the first goal. On the other hand, Japan has never won a world cup match after conceding a first goal. Even though statistics were not on their side, Japan won the match against Germany with a score line of 2-1. After the match, the analytics said that most of the Japanese players had played in the German football cup, which gave them an advantage over Germany.

How can we include crowd support and players' special features into mathematical modeling? I would say it is difficult, which is why sport is such an unpredictable and, thus, fantastic event. We can enjoy sports events only because of their unpredictable nature. If not, sport is not a sport, and let the sport be a sport.

Who is the Next FIFA Champion?

noreply@blogger.com (Dinesh Asanka) — Sat, 19 Nov 2022 18:20:00 +0000

After the Olympics, the world's most famous sporting event will raise its curtain today at Quater. The FIFA 2022 will be kicked off with 32 teams, and the obvious question would be "Who will be the winner". Will it be Brazil, Netherlands, Spain, Argentina, France, England, Belgium, or any other dark horses?

If you remember in FIFA 2010, Paul the Octopus predicted all the winners of the matches that involved Germany and the final between Spain and the Netherlands with an accuracy of 100%.

Paul the Octopus made friends and enemies on his own due to the predictions he made. Now in 2022, the University of Oxford has come up with a mathematical model to predict the winner.

The following figure is its prediction from the round of 16s.

According to the Oxford model, Brazil is the ultimate winner. However, looking at the recent, it will be a surprise.

Further, after 1998 defending champions did not perform well in the world cup finals. So France to be in the semi-finals is something against recent data.

I personally do not believe that predictions can be made for sports concerning its outcome. If you do that, I believe that will be the end of sports. Sport is beautiful because of the extremely high uncertainty. One touch or one move or one wrong decision may change the cause of the game drastically. In 2002 FIFA saw a lot of such matches. Who would have expected the defending champion, France to be beaten by the African nation Senegal? Who would have thought France would be out of the world cup in the first round without any wins? Among all the surprises, South Korea made the headlines by reaching the Semifinals and becoming the giant killer of Poland, Portugal, Italy, and Spain.

Let the sport be a sport!