TEST THE RISK!

Challenges in Evaluating Performance Testing Results

2025-06-20T09:24:00.000-07:00

Performance testing is critical for understanding how applications behave under different levels of load, but interpreting the results remains a complex challenge. Traditional evaluation methods—especially those using binary pass/fail criteria—fail to capture the nuanced reality of modern software systems. As part of Continuous Integration and Continuous Deployment (CI/CD) pipelines, performance tests must provide actionable, reliable insights without manual intervention.

In this post, I’ll share my insights on evaluating performance testing results. It’s the first part of a series aimed at achieving fully autonomous continuous performance testing.

Why Evaluation Is Critical for CPT and Performance Testing

Performance testing is no longer a one-time activity executed before release. With Continuous Performance Testing (CPT), performance checks are embedded throughout the software delivery lifecycle. This integration demands fast, reliable decision-making. But performance data—response times, error rates, throughput, resource utilization—are inherently noisy and multidimensional.

In CPT:

Time is limited. Tests must run and be evaluated quickly.
Automation is mandatory. Manual interpretation won’t scale with frequent code changes.
Confidence is key. Developers need to trust that test results reflect true system behavior.

Therefore, the evaluation process must be intelligent, automated, and resilient to variations in system and workload.

What is Thresholds

Thresholds are the specific values that determine whether a test passes or fails by establishing the minimum or maximum acceptable performance levels. In the context of performance testing, these thresholds are particularly significant because they serve as the benchmark against which test outcomes are measured. By comparing the actual test results to these predefined thresholds, testers can objectively decide whether the system meets the required performance standards or falls short.

Setting Static Threshold for K6

Consequently, setting appropriate thresholds is a critical step in the testing process. These values must be carefully determined, either by analyzing historical data and performance expectations or by calculating them based on the anticipated capabilities of the system under test. This ensures that the thresholds accurately reflect the desired performance goals and provide a reliable basis for evaluating test outcomes.

Thresholds can be established using two distinct approaches:

Static threshold setting
Dynamic threshold setting.

Static thresholds use predefined fixed values, requiring extensive optimization, while dynamic thresholds adjust in real-time based on system behavior analysis. Static methods offer consistency but lack flexibility; dynamic methods adapt but add complexity. Both pose technical challenges, and the choice depends on testing requirements.

The Problem with Static Thresholds

Performance tests often rely on static thresholds to determine pass/fail criteria—for example:

Response time must be below 500 ms
Error rate must stay under 1%
Throughput should exceed 1000 requests per second

Although straightforward to implement, static thresholds face significant challenges. Analyzing complex systems makes threshold-setting difficult, as performance baselines are rarely static. Additionally, evolving codebases and infrastructure require constant adaptation, rendering fixed values impractical over time. Dynamic adjustments may be necessary, but they introduce their own complexities. While easy to implement, static thresholds have major limitations:

Rigidity: Static thresholds do not adapt to changes in workload, system architecture, or infrastructure. A fixed “500 ms” limit might make sense in one deployment but fail arbitrarily in another with slightly different characteristics.
Binary Results: The system either passes or fails, without any contextual interpretation. A 501 ms response time is marked as failure, even if performance is functionally acceptable.
False Positives and Negatives: Minor fluctuations or outliers may cause test failures despite overall system health, or hide genuine regressions if averages remain within bounds.
Lack of Learning: Static thresholds don't improve over time. They remain detached from historical behavior, usage patterns, and test results, which creates missed opportunities for smarter evaluations.

Evaluation Methods for Performance Testing Results

To overcome the limitations of static rules, performance engineers use several evaluation techniques, typically falling into three categories:

1. Static Thresholds

The most basic form of evaluation. Simple rules are defined ahead of time, often as configuration in CI pipelines or monitoring dashboards. Setting static thresholds for performance testing involves defining fixed limits for key performance metrics (e.g., response time, throughput, error rate) to determine whether a system meets acceptable performance standards.

Pros: Easy to implement, understandable for most of the system, but sometimes hard to set the correct values for complex systems. Performance requirements can be directly used if it is already defined.
Cons: Too simplistic, brittle, prone to false alarms.
Best Use: It can be used for quick checks for very stable systems with well-known characteristics.

2. Statistical Methods

These methods treat performance metrics as distributions rather than fixed numbers. They leverage hypothesis testing, standard deviation, and confidence intervals to determine whether current performance deviates significantly from historical norms.

Using Normal-Distribution

For example: Test whether the 95th percentile response time is within two standard deviations of the historical average. Use confidence intervals to detect regressions even if values remain under hard thresholds

Pros: Adaptive, contextual, data-driven
Cons: Requires more data and expertise, sensitive to anomalies. Need to be careful that there might be a trend that every run goes up in the confidence level, so the hypothesis does not fail.
Best Use: Detecting subtle performance regressions or anomalies in CI pipelines

3. Machine Learning Models and Using AI

ML models offer the most advanced and flexible approach. By learning from large amounts of historical data, these models can classify test outcomes, detect anomalies, and even forecast future performance.

Using ML model for Trend Analysis

ML-based evaluation can be applied using:

Supervised learning: Train a model to label runs as “pass” or “fail” based on historical test outcomes and associated metrics.
Unsupervised learning: Use clustering or anomaly detection to highlight unusual behavior without labeled data.
Forecasting: Predict future performance trends using time series models (e.g., LSTM, Prophet).

AI, particularly through ML models, offers vast potential. We can analyze every response in depth to determine the expected value, even when comparing outputs from different endpoints. While countless analyses are possible, the key is identifying the optimal approaches for our system.

Pros: Scalable, context-aware, adaptable to change
Cons: Requires sufficient data, infrastructure, validation, and expertise
Best Use: High-frequency testing environments, dynamic systems, exploratory analysis

Conclusion

The evaluation of performance testing results is one of the most crucial—and most misunderstood—parts of modern testing pipelines. As software systems grow more complex and dynamic, simplistic evaluation techniques no longer suffice.

Static thresholds offer quick wins but struggle with accuracy and adaptability.
Statistical methods introduce nuance and data-driven judgment, ideal for detecting meaningful regressions.
Machine learning opens the door to scalable, intelligent, and autonomous performance evaluation.

To support CI/CD and shift-left strategies, evaluation mechanisms must evolve to be as dynamic and intelligent as the systems they monitor. The future of performance testing is not just about generating metrics—it’s about understanding them in context, automatically, and with confidence.

The Power of Continuous Performance Testing: A Shift-Left and Shift-Right Approach

2024-06-13T11:20:00.001-07:00

no stress we have CPT

The performance of a system can make or break the user experience. Whether it's a website, mobile app, or software application, users expect lightning-fast response times and seamless functionality. As testers, it's our job to ensure that these expectations are met, and one of the most effective ways to do that is through Continuous Performance Testing (CPT). I want to create a blogpost about CPT, which is the main idea of my talk given at ISTSTC-2024. You can reach to my presentation here as well.

What is Continuous Performance Testing (CPT)?

CPT is an approach to testing that involves testing the performance of a system continuously, from the early stages of development to after deployment. It's a shift-left and shift-right approach that enables teams to identify and fix performance issues early on, reducing the risk of downtime, crashes, and other performance-related problems.

CPT is a game-changer in the world of software testing. It involves testing the performance of a system continuously throughout the development lifecycle, rather than waiting until the end to run a few isolated tests. This shift-left approach allows us to catch performance issues early on, when they are easier and cheaper to fix.

But CPT is not just about catching bugs. It's also about optimizing performance and ensuring that the system can handle the demands of a growing user base. By running performance tests continuously, we can identify potential bottlenecks and scalability issues before they become a problem, allowing us to make adjustments and improvements proactively.

monitoring the performance of a system

The Importance of CPT

The traditional approach to performance testing typically involves testing towards the end of the development cycle, which can lead to:

Delayed detection of performance issues, resulting in costly rework
Longer development cycles, leading to delayed time-to-market
Poor user experience, resulting in lost revenue and reputation damage

In contrast, CPT enables teams to:

Identify performance issues early, reducing rework and saving time
Improve system performance, resulting in faster response times and better user experience
Reduce the risk of performance-related problems, leading to increased reliability and uptime

How CPT Works

CPT involves integrating performance testing into the Continuous Integration/Continuous Deployment (CI/CD) pipeline, allowing teams to test performance continuously throughout the development lifecycle.

Here's how it works:

Automated Testing: Automated performance tests are integrated into the CI/CD pipeline, allowing teams to test performance continuously.
Early Detection: Performance issues are identified early, allowing teams to fix problems before they become critical.
Continuous Monitoring: Performance monitoring tools are used to track system performance in real time, enabling teams to identify issues as they arise.
Proactive Improvement: Performance data is used to optimize system performance, ensuring that the system can handle increasing loads and user traffic.

Benefits of CPT

The benefits of CPT are numerous, including:

Improved System Performance: CPT enables teams to identify and fix performance issues early, resulting in faster response times and better user experience.
Reduced Risk: CPT reduces the risk of performance-related problems, leading to increased reliability and uptime.
Cost Savings: CPT reduces the cost of fixing performance issues, as problems are identified and fixed early on.
Faster Time-to-Market: CPT enables teams to release software faster, as performance issues are identified and fixed earlier in the development cycle.

Conclusion

Continuous Performance Testing is a game-changer for teams looking to deliver high-performing systems that meet the demands of modern users. By adopting a shift-left and shift-right approach to performance testing, teams can ensure that their systems are fast, reliable, and scalable. So why wait until the end to test performance when you can do it continuously with CPT?

Comic: https://images.app.goo.gl/TVy8DzqnVN4NhCHA8

Deep Dive into `headless=new` in Selenium

2024-06-12T01:40:00.000-07:00

Introduction

When it comes to automation testing, Selenium is a household name. With its extensive range of capabilities and flexibility, it has become an indispensable tool for quality assurance engineers. One of the most significant features of Selenium is its ability to run in headless mode, allowing testers to execute tests without the need for a visible browser instance. In this article, we'll delve into the world of headless mode, specifically exploring the `headless=new` parameter and its implications for Selenium testing.

What is Headless Mode?

Headless mode is a configuration option in Selenium that enables running browser instances without displaying a visible UI. This mode is particularly useful for automating tasks on cloud infrastructure, where a graphical interface is not available, or for running tests on a continuous integration/continuous deployment (CI/CD) pipeline. By default, when you use headless mode, Selenium will launch a browser instance in the background, allowing you to execute tests without visual rendering.

Chrome --headless old and new

Previously, headless mode was not just a mode but rather considered as another library. This meant that every feature had to be implemented separately for it, leading to a unique set of bugs not found in the Chrome library. Managing this not only demanded extensive effort from Chrome developers but also required users to handle a significant amount of customization for both headless and headed modes. As a result, the current setup illustrates that both headless and headfull modes utilize the same library, ensuring uniformity across functionalities. To delve deeper into the implementation details, you can refer to the blog post created by Chrome developers.

The `headless=new` Parameter

In recent Selenium versions, a new parameter called `headless=new` has been introduced. This parameter signals a significant shift in how headless mode operates. With `headless=new`, Selenium no longer treats headless mode as a separate entity from the standard, full-browser experience. Instead, it leverages the same codebase as the full-featured browser, ensuring that your tests are executed in a more authentic environment.

Why `headless=new` Matters

The introduction of `headless=new` addresses several limitations and drawbacks associated with traditional headless mode. I need to switch to the `--headless=new` because the old version is supporting the testing the Chrome extensions. I can create test cases and run it on headful mode but when I want to run it in headless mode in the CI, it starts failing. There many more benefits and some key benefits include:

Improved Compatibility: By sharing the same codebase as the full browser, `headless=new` ensures that your tests are executed in an environment that closely mirrors the real-world browsing experience.
Enhanced Stability: By leveraging the same rendering engine as the full browser, `headless=new` reduces the likelihood of issues and errors that were previously common in headless mode.
Simplified Maintenance: With `headless=new`, you can write tests that can be executed in both headless and full-browser modes, eliminating the need to maintain separate test suites.

Using headless=new in Selenium with Python

To take advantage of headless=new in Selenium with Python, follow these steps:

Install the required dependencies:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Servicefrom webdriver_manager.chrome import ChromeDriverManager

Import the necessary modules:

options = Options()
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")

Configure the Chrome options:pip install selenium

  service = Service(ChromeDriverManager().install())
  driver = webdriver.Chrome(service=service, options=options)

Create a new instance of the Chrome driver pip install webdriver-manager

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Servicefrom webdriver_manager.chrome import ChromeDriverManager
options = Options()
options.add_argument("--headless=new")options.add_argument("--disable-gpu")
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=options)

driver.get("https://www.google.com")print(driver.title)driver.quit()

Best Practices for headless=new

When working with headless=new, keep the following best practices in mind:Use a recent version of Selenium: Ensure you're running the latest version of Selenium to take advantage of the headless=new features. Carefully consider your test environment: As headless=new uses the same codebase as the full Chrome browser, ensure your tests are designed to accommodate the increased resources required for headless mode. Monitor your system resources: Keep a close eye on system resources, such as memory and CPU usage, to avoid performance bottlenecks.

Conclusion

In conclusion, headless=new marks a significant improvement in Selenium's headless mode capabilities. By leveraging the same codebase as the full Chrome browser, you can now execute tests in a more authentic environment, enjoying improved compatibility, stability, and maintenance. By following the best practices outlined in this article, you'll be well on your way to unlocking the full potential of headless mode in Selenium with Python.

You

AI in Software Testing

2024-06-07T01:50:00.000-07:00

Artificial intelligence (AI) has permeated every aspect of our lives, and software testing is no exception. In fact, AI is revolutionizing traditional testing practices, making them more efficient, effective, and adaptable to the demands of modern software development. By leveraging AI, teams can deliver high-quality software products that meet the evolving needs of both users and businesses. There has been ongoing debate about whether AI will eventually render manual and automated testing obsolete. In my opinion, this prospect is unlikely in the near future, particularly with regards to manual testing. However, AI is making significant strides in automated testing, albeit not yet at a satisfactory level. Currently, the greatest value of AI lies in supporting various activities, freeing up time for testing teams to focus on high-value tasks that require expertise in test engineering and product domain knowledge.

LLM Based Application

If you're a consultant like myself, you may wonder how to harness the power of AI in software testing. To stay ahead of the curve, it's essential to gain expertise in the following areas:

Testing AI-based applications/products

Testing traditional applications is mostly comparing the app with the requirements. However, an AI-based application can create different results with positive test results.
Limiting testing activity is not deterministic for AI testing, understanding the AI algorithm and then doing more testing on the result could open new rooms for testing and it can lead to endless testing effort.
LLM-based application has LLM Models at the heart of the architecture but testing the other parts of the component is similar to testing any products. Here we should add testing the integration of the component. Contract testing can play a crucial role in testing the microservices as well as checking some prompts on the integration of the LLM.

Creating AI-based testing tools to support testing

Based on the experience, we can also create open-source tools to support testing. When we talk about AI in testing, we mostly talk about automation but it is not limited with automation. In fact, many companies created paid tools for test automation. We should leverage the benefits of using AI in other areas as well.

Using AI in software testing like

Automated Test Case Generation: AI algorithms can generate test cases automatically, reducing the manual effort required to create test scripts.
Intelligent Test Prioritization: AI can analyze code changes and prioritize test cases based on risk, impact, and likelihood of failure, optimizing testing resources.
Defect Prediction: AI models can analyze historical data to predict potential defects, enabling proactive measures to prevent issues before deployment.
Anomaly Detection: AI algorithms can identify abnormal behavior in software systems, helping to detect bugs and performance issues early.
Visual Testing Automation: AI-powered tools can automate visual testing by identifying UI changes, layout issues, and graphical glitches across different devices and browsers.
Natural Language Processing (NLP) Testing: AI can understand and analyze natural language requirements and test cases, improving the efficiency of testing processes.
Test Data Generation: AI techniques like generative adversarial networks (GANs) can create realistic test data sets, enhancing test coverage and accuracy.
Dynamic Test Environment Management: AI can dynamically allocate and configure test environments based on project requirements, reducing setup time and resources.
Regression Testing Optimization: AI can intelligently select and prioritize regression test suites, focusing on areas most likely affected by code changes.
Performance Testing Optimization: AI algorithms can simulate real-world user behavior to perform load, stress, and performance testing, identifying bottlenecks and optimizing system performance.
Self-Healing Test Automation: AI-powered testing frameworks can automatically update test scripts to adapt to changes in the application under test, reducing maintenance efforts.
Code Coverage Analysis: AI can analyze code coverage metrics and suggest improvements to ensure comprehensive testing coverage.
Cross-Browser and Cross-Device Testing: AI-driven tools can automate testing across various browsers, devices, and screen resolutions, ensuring consistent user experiences.
Behavior-Driven Testing (BDT): AI can assist in translating business requirements into executable test scenarios, aligning testing efforts with business objectives.
Security Testing: AI can identify security vulnerabilities by analyzing code patterns and behaviors, helping to fortify applications against cyber threats.

Stay tuned, I will be writing more on how we can efficiently use AI in testing. Happy testing!

Enhancing DevOps Efficiency: The Crucial Role of Automated Testing in CI/CD

2023-11-14T05:54:00.000-08:00

In the rapidly evolving landscape of software development, the adoption of DevOps practices has become paramount for organizations striving to deliver high-quality software at an accelerated pace. Continuous Integration and Continuous Delivery (CI/CD) are integral components of the DevOps pipeline, ensuring rapid and reliable software delivery. In this era of DevOps, automated testing plays a pivotal role in enhancing the efficiency, reliability, and overall success of the CI/CD process.

DevOps

Understanding DevOps, CI/CD, and Automated Testing

DevOps: A Cultural Shift

DevOps is a cultural and organizational shift that seeks to break down silos between development and operations teams. It emphasizes collaboration, communication, and automation to deliver high-quality software more efficiently. CI/CD, on the other hand, is a set of best practices that automate the process of integrating code changes and deploying them to production. Continuous Integration involves regularly merging code changes into a shared repository, while Continuous Deployment automates the release of these changes into production.

DevOps is more than just a set of practices or tools; it represents a cultural shift that emphasizes collaboration and communication between development and operations teams. The primary goal is to deliver software faster and more reliably.

CI/CD: The Backbone of DevOps

In the dynamic landscape of software development, the symbiotic relationship between DevOps and Continuous Integration/Continuous Deployment (CI/CD) has become paramount. DevOps practices emphasize collaboration and communication between development and operations teams, while CI/CD pipelines streamline the process of delivering software updates rapidly and reliably. In this era of DevOps, automated testing emerges as a linchpin, ensuring the integrity and quality of software throughout the entire development lifecycle.

Continuous Integration (CI) involves automatically integrating code changes into a shared repository multiple times a day. Continuous Delivery (CD) extends CI by automatically deploying code changes to production or staging environments after passing through various testing stages.

Automated Testing: Ensuring Quality at Speed

Automated testing is a software testing technique that uses specialized tools and scripts to execute pre-defined tests on a software application or system. Instead of manually running tests, automated testing involves the use of automation tools to perform repetitive but necessary testing tasks. The primary goal is to enhance the efficiency, effectiveness, and coverage of the testing process, particularly in the context of software development and quality assurance.

Automated testing is a key enabler for achieving the goals of DevOps by allowing teams to test their code rapidly and consistently. It involves using testing tools and scripts to execute predefined test cases, comparing the actual outcomes with expected results, and providing fast feedback to developers.

Automation should cover all the scenarios that are possible and meaningful, so we can consider the following areas for automation:

Classes/Methods
Integrations

Unit
Component
System

Requirements

Functional
Non-functional

Processes

Development
Testing
Distributing

Implementing Automated Testing in CI/CD

1. Unit Testing

The unit level is the first level of the system and the unit test is designed to check the smallest part of the system. Developers write unit tests to verify the functionality of individual code units. Automated unit testing is the first line of defense, catching issues early in the development process.

2. Integration Testing

Integration tests validate the interactions between different components or modules of an application. The integration level has wider test approaches since the context of the unit is very small and the UI is user-oriented. Everything other than the unit testing are different kind of integration testing so integration testing includes checking the integration of units/components, services, APIs or even systems. Automated integration testing ensures that the integrated system functions as intended.

3. End-to-End Testing

When the level goes upper, concepts of the testing are getting closer to the end-user scenarios so that we can ensure that the end-users do not having bad experience with the product. End-to-end tests simulate user scenarios and interactions with the application. Automating end-to-end testing guarantees that critical user journeys are tested consistently across various environments.

4. Test Environments

Tests should be run against test environments so that tests mimic real users. Establishing automated test environments that mirror production configurations is crucial for accurate testing. This minimizes the risk of discrepancies between testing and production environments.

5. Continuous Monitoring

We can run automated tests whenever possible and meaningful so tests should be run regularly on the production environment to ensure that the prod functioning correctly. Implementing automated monitoring and alerting mechanisms allows teams to detect and address issues in real-time. This proactive approach ensures the reliability of the software in production. Whenever there is an issue with the automated tests, monitoring helps the developer team to

The Significance of Automated Testing in CI/CD

1. Rapid Feedback Loop

Automated tests can be written to all levels so any failures in any levels can raise a red flag. Automated testing ensures a quick and consistent feedback loop. With every code change, automated tests are executed, providing immediate feedback to developers. This accelerates the identification and resolution of bugs, fostering a culture of quick iteration and continuous improvement.

2. Ensuring Code Stability

In a CI/CD pipeline, code moves through various stages, from development to testing to deployment. Automated testing guarantees that only stable and thoroughly tested code progresses through these stages, minimizing the likelihood of introducing defects into production environments.

3. Faster Time to Market

By automating testing processes, organizations can significantly reduce the time required for manual testing. This results in faster and more frequent releases, allowing businesses to respond swiftly to market demands and gain a competitive edge. In today's agile world, we want to deliver small and frequently so there are always packages waiting to be delivered. Each package passes through an automated development process, with appropriate automated tests. These tests make sure that the deliverable is good enough for the production.

4. Cost-Efficiency

Automated testing contributes to cost reduction by decreasing the need for manual testing efforts and minimizing the chances of post-deployment issues. The up-front investment in automated testing tools and infrastructure is outweighed by the long-term benefits of increased efficiency and reduced operational costs.

5. Comprehensive Test Coverage

Automated testing enables the creation and execution of a comprehensive suite of tests, including unit tests, integration tests, and end-to-end tests. This ensures that the entire application is thoroughly examined, enhancing the overall quality of the software.

Challenges and Best Practices

1. Test Data Management

Managing test data is a common challenge in automated testing. Implementing strategies for generating and maintaining realistic test data is essential for accurate testing scenarios.

2. Test Script Maintenance

As the application evolves, test scripts may require updates. Regularly reviewing and updating automated test scripts ensures their relevance and effectiveness.

3. Parallel Execution

To expedite testing, executing tests in parallel is a best practice. This requires scalable testing infrastructure and tools that support parallel test execution.

4. Collaboration Between Teams

Collaboration between development, testing, and operations teams is crucial for the success of automated testing in a CI/CD pipeline. Clear communication and shared responsibilities foster a collaborative culture.

Conclusion

In the era of DevOps, where speed and reliability are paramount, automated testing stands as a linchpin in the CI/CD process. By providing rapid feedback, ensuring code stability, and enabling faster time to market, automated testing contributes significantly to the success of DevOps initiatives. Organizations that embrace automated testing as an integral part of their CI/CD pipeline are better positioned to deliver high-quality software at an accelerated pace, meeting the demands of today's dynamic business landscape. As technology continues to advance, the role of automated testing in DevOps will only become more critical, shaping the future of software development and delivery.

image from globalittrainers

What the Fact with Integration Testing

2023-11-05T02:02:00.001-08:00

Integration testing, a crucial phase in the software development life cycle, plays a pivotal role in ensuring that individual components of a system work seamlessly when combined. Other than the unit tests in the unit level, every thing in the software development process is a kind of integration the pieces. This can be integration-in-small like integration of the components or it cane integration-in-big such as integration of services/APIs. While integration testing is essential, it is not without its challenges. In this blog post, we'll explore the issues of speed, reliability, and maintenance that often plague integration testing processes.

Integration-in-Big

1. Speed

Integration testing involves testing the interactions between different components of a system. As the complexity of the software grows, so does the number of interactions that need to be tested. This increase in interactions can significantly slow down the testing process. With modern applications becoming more intricate, the time taken for integration tests can become a bottleneck, hindering the overall development speed.

Automating integration tests are slow and more expensive because:

Dependent parts must be deployed to test environments
All part must have latest version
CI must run all previous tasks like

Static checks
Unit tests
Reviews
Deployments

OK let's run the API Testing project

Solutions: Employ parallel testing; running tests concurrently can significantly reduce the time taken for integration testing.

Argument: This is OK to be able to reduce the time spent for the integration testing but that is the final step in the process we still need to perform all the necessary steps.

Solution: Prioritize tests; Identify critical integration points and focus testing efforts on these areas to ensure faster feedback on essential functionalities.

Argument: Prioritization should be for any case but we must be careful not miss the necessary test cases. Reporting the issues found for the parts that are other than the essential parts will cause another problem about the overall quality. Also we still have to wait until all the process to be done to run the integration testing again.

2. Reliability

The reliability of integration tests is a common concern. Flaky tests, which produce inconsistent results, can erode the confidence in the testing process. Flakiness can stem from various sources such as external dependencies, race conditions, or improper test design. Unreliable tests can lead to false positives and negatives, making it challenging to identify genuine issues.

Solution: Isolate tests; Minimize external dependencies and isolate integration tests to ensure they are self-contained and less susceptible to external factors.

Argument: Dependencies are the required parts that should be also ready and integrated to the environments. We can reduce the dependencies by mocking but we still need to check the integration of that mocked part of the system.

Solution: Regular maintenance; Continuously update and refactor tests to ensure they remain reliable as the codebase evolves.

Argument: The regular check should be done when ever there is a requirement update and failure of the tests. However we know that the integration testing is isolated from the business after the tests were created and for the most of the case, the integration testing project is driven by the QA team now. For the best case update will be done when the failures occur on the CI pipeline.

3. Maintenance

Maintaining integration tests can be cumbersome, especially in agile environments where the codebase undergoes frequent changes. As the software evolves, integration points may shift, leading to outdated or irrelevant tests. Outdated tests can provide false assurance, leading to potential issues slipping through the cracks.

Solution: Automation; automate the integration testing process as much as possible to quickly detect issues when new code is introduced.

Argument: Automation is in the essence. We have to automate not only the tests but also the process. But the for the most case, integration testing process is not an integral part of the development process so this cause the decomposition.

Solution: Version control; Store test cases alongside the codebase in version control systems, ensuring that tests are updated alongside the code changes.

Argument: Version control inside the integration project helps protecting main branch to be safe but the version of the code that aimed to be tested is not reconciled with the version of the code that is testing. This is not providing extra benefits for the overall quality.

In conclusion, while integration testing is crucial for identifying issues that arise when different components interact, it is essential to be aware of and address the challenges related to speed, reliability, and maintenance. By employing the right strategies and tools, development teams can mitigate these challenges and ensure that integration testing remains effective, efficient, and reliable throughout the software development process. Because of these reasons, it is time to think about "contract testing".

Conclusion

Integration testing plays a vital role in uncovering issues that arise from component interactions, but it is crucial to address challenges related to speed, reliability, and maintenance. By implementing effective strategies and utilizing appropriate tools, development teams can overcome these obstacles and maintain the efficacy, efficiency, and reliability of integration testing throughout the software development lifecycle.

Although contract testing can significantly enhance the testing process, it is important to recognize that integration testing remains indispensable for large, intricate systems with complex component interactions. Integration testing serves as a crucial validation layer in such scenarios, ensuring the seamless functionality of the entire system. By integrating contract testing with traditional integration testing practices, development teams can achieve comprehensive testing coverage, enhancing the reliability and efficiency of their software systems.

Why Contract Testing Matters

2023-02-08T12:15:00.005-08:00

Integration testing has an important role in the test pyramid, it aims to test the integrations of the software products in the integration environments. This means that the software should be built and deployed to an integration test environment and then run the test against the version in the testing environment. If anything wrong occurs as a regression issue then the CI gives feedback about the failures. As a result, the issue should be fixed and built and deployed to the integration test environment. Therefore this process is very expensive and is so late for the modern approach to SDLC, for the shift-left approach. As a solution to this problem, we should consider testing the integration isolatedly within the context of the software itself. Contract testing is an approach to testing the integration depending on the written contract during the build phase. In this post, I want to explain the importance of contract testing.

What is Contract

The contract is a document created by the consumers to list the required functionalities that are used by the consumers. Therefore, the consumer should create a contract for every change in the requirements then, it should be shared with the providers to follow the up-to-date contracts.

What is Contract Testing

Contract testing is testing the integrations of the applications in isolation so is complimentary to integration testing with some extra benefits. Contract testing has a very important role in the software development lifecycle as a good approach to making the integration testing shift left. The contract is a document created by the consumers to list the required functionalities that are used by the consumers. Therefore the contract should be created by the consumer for every change in the requirements then it should be shared with the providers to follow the up-to-date contracts.

There is a misunderstanding about the contract test is rewriting the unit tests against a mock service. But in reality, it is not as simple as this definition since it is bi-directional and it checks the requirement as well as the implementation in build time. The following image also shows where is the contract test in the test matrix. The contract test is basically non-functional testing at the integration level. It is non-functional because we are not aiming to check logic or consumer flows, and also it is checking the integration of the services/APIs so it is an integration level testing.

What is the Problem with Classical Integration Testing

An integration testing project is mostly a project that checks the integration of microservices/services/APIs with some libraries like rest-assured, requests, airbone, etc. Most of the time this project is contributed by the quality team by writing tests for each requirement as soon as the related requirements are turned into functionalities in the product. This is an asynchronous process which means development teams do their jobs and the quality team does their job separately. This causes isolation instead of integration. This is not only done by doing the job but also by running and reporting the results of the tests.

Everyone will be happy until there is a failure in the result of the integration test run. When an error is caught, a hero should be chosen and take responsibility and start investigating the issue. There might have already been changes in the requirement or there is an intermittent issue, so this is most of the time a time-consuming job. Also, there will be job failures until the issue is fixed which will reduce the reliability of the integration test.

Basically, these are the problems of the integration testing:

Flakiness

Data management for each MSs and APIs
Testing different MSs in the same test

Slow

Requires network
Data creation process
Network calls between MSs

Independent from Clients

Tests are not created by clients
Updates are not made by clients
Testing more general instead of the client's requirements

Implementation of Page-Object-Model (POM) to Espresso (Native Android Testing) with Kotlin

2022-02-18T06:46:00.003-08:00

What is Espresso

Espresso is a native test automation tooling developed by Google for Android UI Tests. Espresso is open-source, very stable, and there are many frameworks that have been developed around Espresso. The Espresso testing framework, provided by the Android Testing Support Library, provides APIs for

writing UI tests to simulate user interactions within a single target app.

It has a full feature-set of Junit that works with Espresso
Android Studio provides everything you need
It provides white-box testing
What Google says about it

Page Object Model (POM)

The main idea behind the Page Object Model (POM) design pattern, is to create an object repository for the pages in the application, which will then be used in the tests. In other words, instead of including the page elements and the test code together within the test, we separate them into 2 different entities: the pages and the test scenarios. Using this concept, each screen in the application will have a corresponding page class. The page classes will identify the elements in the page and contain methods that perform operations on these page elements.

This framework adopts the POM design pattern and is structured as follows:

'Pages' folder

Contains all Page object classes.

These classes represent the actual screens of the application. A Page Object class will contain a series of objects, representing the application screen, and a series of methods, representing the features of that screen.

'Tests' folder:

Contains all test classes or scripts.

These classes define the user flow or end-to-end test scenario by referencing the page object class methods.

'Utilities' folder:

Contains helpers, extensions to the Espresso, and scripts.

How to Write Tests

All page object classes and tests will extend the BasePage or BaseTest, thus inheriting all the base methods. These classes contain common functionality for e.g. setting up the app, terminating the sessions, or initializing page objects.

1. Create the BaseScreen

2. Create a Screen class named as DictionaryScreen by extending the BaseScreen. Define UI elements via the app using the accessibility identifiers of the elements

3. Create the BaseTest. The BaseTest should handle all the basic things for launching, terminating the app also it should include creating basic test objects for writing tests.

4. Create a test class for the dictonary screen named as DictionaryScreenTests by extending the BaseTest and calling them for the test steps. Test setup should handle all the required actions

Do you want to run these tests in your local and have experience with Espresso for UI tests, Junit test for unit tests, and also overall Android app structure then you are good to go with my repo here: testdictionary-android-app.

Implementation of Page-Object-Model (POM) to XCUITest (Native iOS Testing) with Swift

2022-02-07T22:51:00.003-08:00

What is XCUITest and XCTest

Xcode UI Testing is also known as XCUITest is a huge expansion of the testing technology in the Apple
platform apps like iOS.XCTest is a general testing framework, used for unit integration, network, or performance testing of Apple platform apps. With XCTest you can treat it as a xUnit type of test framework, which can be used for lower-level testing.

The XCUITest framework is an extension of the XCTest framework but there are some basic differences between XCTest and XCUITest framework.

XCTest framework is a completely white-box framework where you can access the data and API in your main app.
XCUITest is completely black-box to your app, so you cannot access the data or API of your main app.

Page Object Model (POM)

This framework adopts the POM design pattern and is structured as follows:

'Pages' folder

Contains all Page object classes.

'Tests' folder:

Contains all test classes or scripts.

These classes define the user flow or end-to-end test scenario by referencing the page object class methods.

'Utilities' folder:

Contains helpers, extensions to the XCTUITest, scripts.

How to Write Tests

1. Create the BaseScreen

2. Create a Screen class named as HomeScreen by extending the BaseScreen. Define UI elements via the app using the accessibility identifiers of the elements

3. If the element doesn't have an accessibility element then find the relevant view and add the accessibility identifier for it. ex: `homeText` that is used `readmeMd`. This is an example of why XCUITest is not only a black-box solution. It enables us to modify source code for the sake of writing better-automated UI tests. This is one of the advantages over the non-native automation toolings.

4. Create the BaseTest. The BaseTest should handle all the basic things for launching, terminating the app also it should include creating basic test objects for writing tests.

5. Create a test class for the screen by extending the BaseTest and calling them for the test steps. Test setup should handle all the required actions

Do you want to run these tests in your local and have experience with XCUITest for UI tests, XCTest for unit tests, and also overall iOS app structure then you are ready to go with my repo here: testdictionary-ios-app.

Testing Mobile Apps in the Cloud - Product Comparison

2022-02-07T10:12:00.001-08:00

Cloud testing is raising, especially for mobile automated testing it is even going faster. Basically, cloud testing means that an application is installed on real devices or simulators/emulators that are hosted in one of the cloud testing services providers. These devices are maintained by the service providers and promise to be up and ready for service with a subscribed contract so no need to tackle these kinds of stuff for the sake of focusing on building the right product with well-tested features. In this post, I want to share some info about main mobile cloud solutions in the market.

PRODUCT	PRICE	SUPPORTED TOOLS
AWS Device Farm	Pay as you go $0.17 / DEVICE MINUTE Test on any of our devices in parallel and pay just for what you use Your first 1000 minutes are free* Unlimited testing STARTS AT $250.00 / MONTH Test as much as you want each month for a flat rate Private devices STARTS AT $200/MONTH Test on dedicated devices deployed exclusively for your account	Appium Java (TestNG) Currently, Device Farm supports Java 8 for running Appium Java tests. Java (JUnit) Currently, Device Farm supports Java 8 for running Appium Java tests. Node.JS You can use any Node.js version when you run Appium Node.js tests on Device Farm. Specify the Node.js version in the test spec YAML file. Python In standard mode, Device Farm supports Python version 2.7 for running tests. In custom mode, both Python 2.7 and Python 3 (3.7) are available. Ruby You can use any Ruby version when you run Appium Ruby tests on Device Farm. Specify the Ruby version (for example, 2.5.1) in the test spec YAML file. Calabash Espresso XCUITest UI Automater
Firebase Test Lab	Free Virtual Device Tests $1/device/hour Physical Device Tests $5/device/hour pay as you go Virtual Device Tests 10 tests/day Physical Device Tests 5 tests/day	Test Tools Espresso UI Automator 2.0 XCUITest
App Center / Visual Studio App Center	30 days free trial For 30 days, you will be able to test your app on more than 450 device configurations. With the trial, you can test your app on 1 test device concurrency with unlimited device hours. Run unlimited, faster builds Ship your app faster with multiple builds running in parallel. Each build you pay for comes with unlimited build time, ensuring all builds run and complete. $40/month per build concurrency Standard Test Plan Each device concurrency comes with 30 device hours. $99/month per device concurrency Enterprise Test Plan Unlimited test hours on thousands of real devices and OS configurations. Each device concurrency comes with unlimited device hours. $499/month per device concurrency	Test Tools Appium Espresso XCUITest Calabash - no support anymore
Browserstack	App Automate 199$ / month - 1 parallel 399$ / month - 2 parallel 999$ / month - 5 parallel Live Testing 29$ / month - 1 parallel 99$ / month - 2 parallel 199$ / month - 4 parallel	Test Tools Appium Espresso XCUITest
SauceLabs	App Automation 399$ / month - 2 parallel Unlimited automated testing minutes 799$ / month - 4 parallel Unlimited automated testing minutes Live Testing 49$ / month - ? parallel	Test Tools Appium Espresso XCUITest
Kobiton	App Automation 40$ / month Unlimited parallel 500min /month 167$ / month Unlimited parallel 2000min /month Live Testing Included	Test Tools Appium Espresso XCUITest Scriptless automation

Let's look at the high-level to detail by mindmap picture

Bitrise integrate Browserstack to Run XCUITest

2022-01-21T03:32:00.006-08:00

Bitrise is one of the best solutions for mobile application development. It gives us many mobile app-related steps to create CI/CD (continuous integration / continuous deployment) pipelines (or workflows as in the Bitrise). Around the mobile application development ecosystem, we have many cloud solutions to make quick and reliable development processes. Browserstack is also one of them, it is widely used in web development and testing but also it also has nice features like 'App Live' and 'App Automate' for mobile app testing.

In this post, I want to share how we can integrate Browserstack to Bitrise to run XCUITest.

Run XCUITest on Browserstack via Bitrise

If you want to run XCUITest on a real device then you must build the for generic iOS devices. Building for generic iOS devices requires provisioning so you should ensure that you added the provisioning step to the workflow. Also to be able to run an iOS build on a real device, the device should be registered to the Apple console. This is required by Apple security concerns. However, device farms Firebase Test Lab and Browserstack handle this issue, so no need to be concerned about the limit of registered devices.

In this example, we are using a script to handle all the staff for us. So it does the followings:

Create a .ipa file from the derived artifact
ZIP the test file - UITests-Runner.app to UITests-Runner.zip
Upload the app - .ipa created at Step-1
Upload the test file - .zip file created at Step-2
Trigger test with uploaded files and desired devices and test options
Wait until tests to finish and evaluate the result

For the script, all the required files are created by the step "Xcode Build for testing for iOS" so this step be set up correctly. We are using file under "$BITRISE_TEST_DIR_PATH"

See the script to run XCUITest on Browserstack

Bitrise Integrate Browserstack to Run Espresso Tests

2022-01-21T03:20:00.002-08:00

In this post, I want to share how we can integrate Browserstack to Bitrise to run Espresso by directly calling the Browserstack API.

Run Espresso on Browserstack via Bitrise

If you want to run Espresso tests on a real device then you must build the android project testing so we need to add the 'Android Build for UI Testing' step then add a script runner to run the Browserstack script. See the workflow:

In this example, we are using a script to handle all the things required for running the Espresso tests on Browserstack. So it does the followings:

Upload the app - debug.apk created by "Build for Test" step
Upload the app - debug-androidTest.apk created by "Build for Test" step
Trigger test with uploaded files and desired devices and test options
Wait until tests to finish and evaluate the result

For the script, all the required files are created by the step "Android Build for UI Testing" so this step be set up correctly. We are using files that are added as environment variables: BITRISE_APK_PATH and BITRISE_TEST_APK_PATH

See the script to run the Espresso tests on Browserstack

Benefits of Using Native Mobile Automation Tooling: Espresso and XCUITest

2021-11-22T08:43:00.001-08:00

Testing will be more efficient when testers and developers work in collaboration. Especially when it comes to automated testing, we have many challenging points that should be solved with the help of the other team members. This is a basic explanation of the cross-functional team. For mobile application automated end-to-end testing, this cross-functional team approach can be easily applied by contributing the same code base so native toolings give us this. This can increase the quality of the tests code and the framework by the feedback from the developers, and increase the quality of the tests by the feedback from the testers.

In this post, I want to give some idea about the benefits of using native mobile automation toolings such as Espresso and XCUITest.

What is Native Tooling

Native mobile automation toolings are integrated libraries to the mobile application development framework so that developers can write UI tests for the products under development. Native toolings are part of the development frameworks so it is always guaranteed that the tools are always up to date with the development frameworks and upcoming new features. Native toolings use the same development languages as the development frameworks so the contribution to tests and the application development would be similar approach and strategy.

Since we are talking about two major mobile device operating systems, we are just focusing on Android and iOS. Native mobile automation toolings are

Android - Espresso
iOS - XCUITest

How Native Toolings Work

Native tools work natively, which means that they are providing direct methods via APIs that are done inside the development project globally. Interactions are made by directly calling the UI objects inside project architecture by the APIs.

So these are the main benefits:

APIs know the state of the objects
We are free to use tools as black-box as well as contribute the development code as white-box
We are free to reach the UI objects
If the app is the Matrix, native tools are the agents inside the matrix

How Espresso Works

Espresso is a test automation tool developed by Google for Android UI Tests. Espresso is open-source, very stable, and there are many frameworks have been developed around Espresso.

It has a full feature-set of Junit that works with Espresso
Android Studio provides everything you need
It provides white-box testing
What Google says about it

This is a basic implementation for Espresso. You can see source code, unit tests as well as UI tests. For more information about the implementation, please check this project.

How XCUITests Works

XCUITest stands for Xcode User Interface Testing and it is UI automation framework for iOS applications. XCUITest is developed on top of XCTest and Accessibility. It is launched by Apple in 2015. It supports all iOS functionality in terms of testing them by white-box testing techniques.

XCUITest is native and inside the app source code with different target
XCode provides everything you need
It provides white-box testing
XCTest provides everything you need for testing capabilities
Accessibility provides UI interaction for all kinds of XCode product

This is a basic implementation for XCUITest. Again you can see source code, unit tests as well as UI tests. For more detail about the implementation please check this project.

Benefits: Handling Objects

Native toolings are inside the projects so no need to worry about the use of external tool to get the related objects.

Espresso and XCUITest provides the following benefits

APIs provide methods for all kinds of object
Methods are always up-to-date with the latest improvements/updates
Freely update the source code for the sake of UI-test

Benefits: IDEs are Friendly

Android studio or the XCode comes with development experience so we use these experiences for the development of the test. Dependency management inside the projects IDEs allow us to write, run, debug the tests. Debugging for both android studio and XCode provides a great experience for real-time test writing.

Benefits: Servers -- NO

Performance is very important when you grow and have to handle hounderents tests. The main point for the performance is how much time your tests require for running them locally and in cloud services

Simple UI frameworks interact with the objects so no need to translate with servers
Native tooling know the state of the elements so no need to check them regularly
Automation code is inside the test package so no need to handle it differently while testing on the cloud. This reduces the time drastically

A simple test to compare the performance of the tools shows that native toolings are always better than Appium. This was done by TestProject engineers, so for more detail, please read this post.

Benefits: Team Building Around Quality

Better performance from an automation project can be only get when the developers and testers work cooperatively. Developers are always result-oriented people so they focus on the development of the features, but the QAs/SDETs must be keen on the quality. This quality not only the product quality but also the quality of the tests, process, and documentation. This can be achieved by cross-functional team building.

Cross-functional-team requires the followings:

Team spirit
Shared common knowledge, every team member should know basics about

What is being built
How things are developed
How things are being tested

The absence of one person from a role can be filled by another role

QA/SDET can be a developer
Developer can be QA/SDET

Centralized Monitoring System for Automated Tests with Elasticsearch and Kibana

2020-06-23T08:15:00.002-07:00

In this post, I want to give you some practical information about a monitoring system for automated tests. I will touch the following topics:

Why we need a monitoring system for automated tests
What are the potential benefits of the monitoring system
A simple approach using the following technologies:

Elasticsearch
Kibana
Firebase test lab, gcloud and gsutil
XCTest and Espresso

What is the Centralization of Log Systems

In a development pipeline, we have a variety of tools and technologies with different languages and scripts. CI/CD pipeline is a good place the see the process of the development, but to get the detail of an item in the pipeline, we need to get into that item and tackle with logs and data. Some times this investigation requires some capabilities to understand it. One of the easiest ways to get a common understanding from the logs in the pipeline is using one easy tool for the whole process. When we collect all the logs and data into one place then we can start debugging activity from here. Therefore the centralized log system means that all the logs and data related to the development pipeline should be stored in one place with a timestamp and/or tags for data separation.

What are Automated Tests

Any types at any testing level which are scripted and run over tools can be called as automated tests. Therefore, it is not aiming to one level just as a unit test or an integration level testing and not also functional or non-functional tests, any tests that we can run on CI/CD pipeline could be considered in the fashion. Basically, the following test matrix shows the test types considering the test levels. As the test pyramid suggests that the more test in the low-level, the more reliable and fast automated tests we would get so staring from the unit level most of the tests could be automated and bonded to the monitoring system.

The test types considering the test levels

What Means the Monitoring System for Automated Tests

As we can see from the testing types table, we have lots of testing types and each testing types have a great variety of tools with different language support. Each test type produces different kinds of output, such as XML, JSON, HTML, and so on, in a different environment so reaching these outputs and understanding them is a pretty hard and time-consuming task and it also requires credential management for each environment.

The monitoring system for the automated test can collect the test results from the different environments as soon as the tests completed or during a test run. Then the data can be used to draw graphs, create tables, and some simulations. In this way, everyone can see the result of all the automated tests written for the product in realtime.

Why We Need a Monitoring System For Automated Tests

As defined in the previous section, collecting the logs and data into one common place makes it easier to work with these data. These data can be used for monitoring, debugging, reporting, alerting, and other purposes so important thing is collecting data and making them available for the whole team.

The benefits of the monitoring system for automated tests can be summarized into the following topics:

1. Team Awareness

If everything in a system is transparent, then the system itself shows what it is. There will no extra word to explain it. Every day the monitoring system shows the result of the tests run against the product and it highlights how much of them passed and failed. If there are two kinds of tests on the monitor and one has a red part, then there is no more word to explain that the red part test has a problem and it needs to be solved. This can create team awareness of the product to take any action about failure.

2. Contribution

Anyone can write a test for any test type if he is willing to write. The monitoring system can be itself a self-explained system that you can easily see the separation between test types and products. If the number of unit tests written for the Android app is half of the number of the unit tests written for the iOS app then the team should take action to remove this gap.

3. Quality-Centric Development

Software development is a continuous effort so every commits adds something new on the product. If there is something wrong with one of the quality metrics then the related action should be taken to improve the related metric.

4. Increased Number of the Automated Test

The stability in the results is good as far as the numbers of the tests go up. If the number of the test stay stable for every commit, we will get lower code coverage. Each new requires new automated tests and fixing defects also requires updating correctness of tests or adding some extra tests for edge cases. Monitoring these metrics, courage us to write more tests and make them passed.

5. Complete-Working Process - CI

In practice, sometimes, the CI process doesn't have tests, or these steps are somehow skipped. The main responsibility of the team is to develop a high-quality product. The meaning of the high-quality product depends on the requirement but there is only one way to understand the quality of the product is to check requirements, or other meaning testing it against to the requirements. The good news is that; if we can monitor the tests, we should have already implemented those tests to the CI.

6. Easier Debugging

The best part of the centralized logging system, having a starting point for debugging errors, failures, exceptions. Logging not only results but also logging the deepest info of the failures and the exceptions make debugging easier.

What is Elasticsearch

Elasticsearch is the heart of the Elastic (ELK) Stack. Elasticsearch is an open-source search engine. It is built on Apache Lucene. It provides a Restful API. The first version of Elasticsearch was released in 2010, then it became the de facto search engine and it is commonly used for full-text search, business analytics, log analytics, security intelligence, and some of the operational intelligence.

Elastic Stack

Elasticsearch is a full-text and distributed NoSQL database so it stores documents with indexing them instead of tables and schemes. We can send data in JSON format to the Elasticsearch database by Restful API or Logstash, another tool for logs operations.

What is Kibana

Kibana is a product from ELK Stack. It is an open-source data virtualization tool that is designed to use Elasticsearch as a data source. Once you integrated Elasticsearch and Kibana and fed the Elasticsearch database with data then you use the Kibana. Kibana has a user-friendly web user interface to create indexes, search, and dashboards. On the dashboard, you can add flexible chars, graphs, heat maps, histograms.

Setting Up Elasticsearch and Kibana in Docker

Containers make everything easier, we can run Elasticsearch and Kibana easily with a docker-compose.yaml. Basically, I am using the docker images provided by Elastic company officially with the version of 6.6.0 for both Elasticsearch and Kibana. Then it runs the Elasticsearch on port 9200 and the Kibana on port 5601. Therefore after the docker-compose up command, you can navigate to http://localhost:9200 for Elasticsearch and http://localhost:5601 for Kibana.

Export Data from Firebase to Elasticsearch

I mentioned above, to feed the Elasticsearch database Logstash is used for most generic architecture. However, in this post, we want to integrate mobile application testing in the cloud. We use Firebase as a cloud mobile testing provider. Firebase provides test labs to run mobile application testing. For mobile application testing we have implemented tests with native tools so for Android testing, we will use Espresso and for iOS testing, we will use XCTest.

To be able to export data from Firebase to Elasticsearch we need to follow these steps:

Run mobile tests with auto-generated UUID by using GCLOUD cli
Calculate the related test result files with this UUID
Grasp the .XML files from Google Storage by using GSUTIL
Export data on those .XML files to Elasticseach by using Restful API. For each run to be distinctive, create new label called timeString. This variable will be used to separate the test runs and make them ordered.

You run the script with android or ios options:

# run test and export data
bash run_tests.sh android
bash run_tests.sh ios

# only export data
python3.6 export_xml_data_elastic.py android
python3.6 export_xml_data_elastic.py ios

and this is the Python script to export the data

Creating Interactive Dashboard on Kibana

Search on Kibana with Filters

Go to the Discover menu and select one item in the list then click the + button of the related filter item. Once you click the + button next to the name item it filters the search item with value name has and then I want to add the testProject to filter both ios project with unit tests.

Kibana searching with filters

Create Visualizations

Click Visualize then click + button on the middle of the page, it will pop-up types of visualization menu then you can select one of your favorite charts. Then it opens the saved search item, you can select the search you saved in the previous step.

Kibana setting the metrics

I choose the bar chart, later I will add one column to show the number of success, failure, and error. As you can see from the picture above you need to define the Metrics and the picture below shows how to split the chart for each run by setting the Buckets.

Kibana setting Buckets

Creating an Interactive Dashbord

Finally, we can create a Kibana Dashboard that can display every chart related to the automated test results. Click on the Dashboard menu and click the add button on the top-right then select the visualization that you created in the previous step. That's all! Now we can check the dashboard.

Kibana dashboard with the result of automated tests

If you want to see the repository and contribute it, you can fork it from Github :)

(Framework) API Test automation with Ruby Rspec and Airborne

2020-05-11T08:14:00.001-07:00

Automated tests for API and services are very important so there are lots of solutions for API test automation in the market. These solutions can be commercial-off-the-shelf (COT) or opensource libraries also, there are many frameworks created around these opensource libraries. Testing APIs with tools that have IDE can be easy to start and learn but it can be challenging to customize for your unique problems or integration to CI/CD tools can be hard. Therefore I can suggest you use one of the opensource libraries by implementing it with your favorite language. In this way, it can run faster, easily extendable, and configurable. I have created a framework with Airborne and Rspec for automating API tests, you can check it from Github. In this post, I want to explain what is Airborne and how I implemented a framework around it. You can simply update the environment variables and write your specs file in the `specs` folder.

What is API Testing

API stands for an application programming interface. APIs are providing an interface of your system to the world. API's endpoints can be called by requests and these calls return relevant data. An API can be considered as a source for clients such as web, android, ios, desktop applications, or even other APIs. Testing APIs from these clients will be difficult, expensive, and slow, therefore it is very important to test APIs as soon as any changes are introduced in the codebase.

API testing includes calling endpoints of APIs with relevant data and credentials. Each call should be asserted in terms of validity and verification. Some of the scenarios can be end-to-end tests, which means that you can not test these scenarios in one single call, instead, you need to call several API calls in accordance with your user-scenarios. All these works called API testing. API testing can be performed manually with some tools or you can write your automation project and run them automatically. In terms of `exploratory testing`, you should adapt your testing tools to perform manual testing for exploration.

What is Airborne

Airborne is a GEM created to test APIs on top of the Rspec and RestClient. Airborne provides a framework to write automated tests for APIs. There are many tools and libraries created for API test but Airbone is Ruby equivalent and it provides a simple response check so that you can whole JSON response at on with expected data or type of data. It also uses the RestClient for handling HTTP requests, which is very mature and reliable. If you have a struggle using Airborne you can call RestClient directly in your test either. The bad part is that Airborne doesn't have good documentation but once you start writing simple tests it becomes easier.

How to Write Tests

Airborne is an abstraction layer on top of rest_client, it makes testing easier with providing simple syntax. Airborne provides basic HTTP methods such as get, post, put, patch, delete, head, and options requests. You can use any of these methods to write your tests, see the example below:

Testing needs assertion, for API testing assertions are written to check the object types, object values in the JSON. Airborne has built-in assertions to check the JSON effectively. These are the Rspec based asserting functions:

expect_json_types - Tests the types of JSON property values returned
expect_json - Tests the values of the JSON property values returned
expect_json_keys - Tests the existence of the specified keys in the JSON object
expect_json_sizes - Tests the sizes of the JSON property values returned, also test if the values are arrays
expect_status - Tests the HTTP status code returned
expect_header - Tests for a specified header in the response
expect_header_contains - Partial match test on a specified header

TestOps: What is it and Why we need it

2020-05-10T18:19:00.002-07:00

Years of experiences in the software industry shows that there are many more thing that has not been discovered yet. Every approach found in the software industry aims to accelerate the development speed with more customer satisfaction. Hence the new term "TestOps" is also creating different perspectives on the relationship between test activities to the operational activities within the DevOps culture. In the post, I want to explain my experience with the TestOps.

What is TestOps

It shortly stands for tests and operations. Within the DevOps culture, it is a sub-discipline DevOps. There are two approaches to TestOps, these are related to the test approach that you are applying the overall test approach. Mainly if you are applying `shift-left testing` then the TestOps should be adopted to have more collaboration with the development team, `TestOps shift-right`. On the other hand, if you are applying `shift-right testing` then the TestOps should be adopted to have more collaboration with the operation team, this can be called `TestOps shift-left`.

Why We Need TestOps

Developing something from idea to a product is needing different expertise such as product owner, developer, tester, DevOps. This shows that one can not handle everything by himself, or doing one responsibility doesn't mean that you are the expert and doing the correct thing. In the testing, there are always changes. We need to adapt ourselves to the waterfall, v-model, agile development process, and also application architectures patterns like monolithic, microservices. Required tests for each process and pattern are different so the required capabilities of the testers are also different. The emerging pattern nowadays is microservices and the most popular development process is agile.

With emerging trends in the microservice architectural pattern, shift-right testing has risen significantly. One of the pioneers who applied the microservice pattern widely is Netflix, and they are suggesting to test those hundred of microservices in the later stage of development as well. Let's look at the real-world example of Amazon and Netflix below:

Number of Integration Point For Amazon and Netflix

Since there are hundreds of microservices and the number of integration points between each of them is very high. Let's just think 2 services then the integration points become 2. If there are 3 services, then it becomes 3. Respectively 6, 10, 15, 21, and so on. It increases drastically when the number reaches to hundreds, for example, 100 services have 4950; 500 services have 124750 two subset integration points. Basically, the combination formula is below, where the n is the number of microservices, r is subset/subgroup which is 2 for integration calculation.

Netflix, as well as Google and Amazon, also has been the pioneer in the testing of the Microservices. Thanks to Netflix, they explained the experience transformation publicly. Netflix and also Spotify have changed the use of the traditional test pyramid. It is reverted and it becomes a test diamond, which is saying that the unit test is still important but instead of writing exhaustive unit tests we should focus on integration test more. For detail how Spotify change the test pyramid for microservice testing you can read this post.

Traffic in Netflix

So far I want to explain the system that we need to make an impact on the quality of the overall system becomes more and more complicated. Handling the test activities of those systems should be updated with new approaches. Whatever your testing approach is shift-left or shift-right, you should use the discipline of TestOps effectively.

TestOps shift-left

TestOps is needed when you want to leverage the test automation benefits. This requires lots of expertise for TestOps shift-left, these are some basic tasks that you will need to do for a good test automation practice, such as:

Using containers
Using container orchestration tools
Creating test pipelines with automation tools
Integrating test tools the pipeline
Integration test pipeline to the main development pipeline
Creating or using a well-known automation framework
Writing deterministic tests
Creating test data
Removing dependency in the tests
Creating isolated test environments
Running test parallel
Creating test reports
Binding these test reports
Rerunning failed tests
Extracting flaky tests and making them investigated
Destroying the test environments after tests
Removing test data
Creating a monitoring system for the visibility of tests
so on ...

TestOps shift-right

Let's look at the TestOps shift-right, which is mostly focusing on the collaboration with the operation team. Even though it includes most of the items for TestOps shift-left since it requires continues testing practice, its requirements are also different and we can list them as:

Having continues testing
Testing on the live environment
Effective use of a monitoring system
Leveraging data on the live environment to grasp anomalies
Creating dashboards/charts from the live data
Chaos engineering, defined by Netflix. Basically, identifying failures before an outage happens by injecting controlled failure scenarios for the reliability of the system.

Where We can Apply TestOps

We can see the benefits of TestOps in some situations more such as performance testing, test automation. For a well-designed performance testing, the strategy should include the real user scenarios. To be able to draw these scenarios, we need to tackle the real user data which is collected from the live environment. Let's list the items needed for performance testing:

Real user flow data should be collected
Data should be analyzed to create a distribution table for each user flow
This distribution should be converted to a percentage of usage
Test scenarios should be created depending on the real users
Each test scenarios should be weighted with the percentage
The test environment should be created if necessary, most of the time live environment should be used
Environments for running performance scripts should be prepared, use the cloud services. If the master-slave configuration needed, the environment for each slave should be prepared. The better way is to create an IAAC file to handle this creation and scalability automatically.
Prepare monitoring tools
Run the tests
Check the monitoring tools if loads are received correctly
If any failure occurs, reconfigure the test/live environment and re-run the script. If everything goes well, stop the performance scripts
Collect the data
Stop every environment
Create a report
Possible I missed something

Since these activities are not done by a single person, TestOps discipline helps to handle these tasks. In the same way, test automation also includes most of the technical expertise to leverage the benefits.

How We can Align to TestOps

TestOps requires some new qualifications for QA/Test Engineers such as learning how DevOps tools can accelerate the testing experiences and also how the QA team helps the operation team to get better customer experiences. Let's summaries the qualifications which are needed to align TestOps discipline:

Learn DevOps tools

Containers
Container orchestration tools
Cloud testing ability

Learn scripting languages
How to spot defects
How to debug defects
More automation, not only test but also process
Apply continues testing
Adapt yourself to new tools and technologies
Monitoring tool experiences

Search data
Create charts/graphs
Create reports

Learn how to communicate with the operation team

I have given a talk about What is TestOps in Five Question, you can it watch it (in Turkish)

GitHub Actions: CI / CD Tools - Applying for Ruby / Rspec Project

2020-05-02T08:19:00.003-07:00

Github released the beta version of "actions" last year for creating a CI/CD pipeline for the projects. This feature enables us to create a pipeline easily with the web interface of Github. I want to run the tests for one of my public project in the Github before merging branches to the master. For testsozluk aka "test dictionary", which contains terms frequently used in the software industry with the meaning and usage of them in Turkish. For the project, tests are critical because JSON files in the Github project are the data source of the application so if any corruption in the data may break the android and the ios application. The tests in the project are written in Ruby by using Rspec to assert them so I used the Github actions to verify before merging code. In this post, I want to explain how this can be done.

Github's actions are very easy to apply. You can add it just go to the Actions tab in your project and then select one of the predefined options on the list. For Ruby projects, there is one for Ruby project and one for Ruby Gems project, or you can skip these predefined workflows and create your own workflow with create yourself option. Ruby Gems project is also including deployment part so you can easily push your gem to RubyGems with the credentials easily no need to push it in your local repository. Let's look at the Ruby projects, see the image below:

When you get the predefined Ruby workflow, you can add some tasks related to your pipeline and customize it. Basically, the workflow that I want to apply is to create a virtual Linux machine and install Ruby with the version of 2.6 and install requirements with the bundle. This workflow will run on push, but you can also set it on pull_request, or on schedule by defining a schedule with Cron syntax. It also has a webhook to trigger the workflow. I selected ubuntu-latest as a virtual machine but there are also other options as following:

runs-on:

ubuntu-latest, ubuntu-18.04, or ubuntu-16.04
windows-latest or windows-2019
macos-latest or macos-10.15

For the project, tests are present in the `src/test` folder since I created the tests with RSpec it has its own structure, so I need to run tests in the test folder. Therefore running tests should have two steps; first I need to go to the test folder and then run the RSpec specs/test_spec.rb command to run them. The run command in the workflow supports multiple commands with adding pipe |. Let's look at the workflow as below:

Basically, I have created three tests to check first if the file is well-formatted JSON, second every term has term and meaning fields, and third there is duplicated term. If one of the tests fails, it doesn't allow us to merge it. See the tests:

To be able to stop merging to master when the tests are failing, you should add a rule in the Setting tab of the project and select the Branch option in the right list and add a rule that you should enable "Require status checks to pass before merging" and check the test which is a given name in your workflow yaml file. With this setup, merging action is now dependent on the result of the tests.

Let's check the failing and the passing scenarios:

Automated Performance Testing with K6

2019-12-31T06:12:00.000-08:00

Automated testing is running the tests and reporting results by tools with scripts. Automated performance testing is designing the performance test scenarios to run them spontaneously by tools and evaluating the result by tools to decide to go or not to go further.

Automated Performance Testing is

A subset of the performance test scenarios

Designed to run spontaneously by tools

Designed to evaluate the result by tools

Designed not to break the system

Having decision metrics, ex: max response time, max-average resp time, ...

Designed to create reports for the concurrent and historical run

What Types of Performance Testing Should be Automated

Automated performance testing can be applied to all types of performance testing. However, each type needs a different level of maturity and sanity.

Automated Performance Testing is

Easily applied to load testing

Hard to apply stress test, spike test, but have benefits

Very hard to apply Soak-Endurance test

Automated Load Tests

Since the load test is aiming to the response of the system under a specific load and it is a success belongs to the pre-defined metrics, load testing is the best-suited performance testing type.

Automated Load Testing is

Needing a specific load level

Needing pre-defined metrics

Needing evaluation of the result

Needing exactly the same environment - only one thing can be testable

Needing tools and script knowledge

Easy to integrate CI/CD pipeline - everything is scripted

Reporting the performance of the system when something changes

Tools Needed for Automated Performance Testing

Performance testing depends on two main parts. The first part is creating loads and the second part is using these loads for test scenarios. When it comes to doing automated performance testing, we need more tools

Tools Needed for Automated Performance Testing are

Performance Testing tools

Open-source

Jmeter - Apache project, v1.0 1998
Gatling - Enterprise support by Frontline
K6 - Enterprise support by LoadImpact
Locust
and many others

Commercial

LoadRunner by Microfocus (previously Mercury - HP)
Blazemeter - support multiple opensource tools
LoadUI Pro by Smartbear

Databases

InfluxDB - Opensourced times series database
MongoDB - Opensourced document-based database

Reporting tools

Grafana
Kibana
Graphite

Orchestration Tools

Docker Compose
Kubernetes
Prometheus
Mesosphere DC/OS
And others

Automation Tools

And others
Jenkins
TravisCI
TeamCity
CircleCI
Code Ship
Gitlab CI"

A Way of Implementation of K6 for Automated Performance Testing

K6 is an open-source performance testing tool. It is designed for developers, it gives us many good features to run tests in a safer, faster, easier integrated way. Very easy to integrate it to CI/CD.

Some of the Good Features of K6 are

Precious in performance, among the top 5 in comparison

Written in Golang, allows writing test script in Javascript

Already dockerized

Easy to connect to DB (such as InfluxDB)

Easy to create reports (Grafana)

Has build-in Thresholds

And many others

K6 has rich libraries that you can use in the script. The request is sending by HTTP from k6/http. Grouping is letting you run the same kind of request in a group. Check the simple test scripts which have get requests to the three URLs and post requests to register endpoint with dynamic data.

Command-line supports many parameters to run your scripts. For a load test, we have fixed virtual user and fixed test duration so the run script should look like

For a spike test, we should load the system with a nominal load and the load should increase suddenly. Let’s focus on at the --stage option for this.

K6 can be easily integrated with an InfluxDB instance so in the setup we are also running the docker-compose to create an InfluxDB instance with will use for feeding Grafana dashboard to visualize all the data.

The K6 sends the results to the InfluxDB, we can virtualize this data with Grafana. You can use the dashboard created by the LoadImpact

Add these scripts to your CI/CD pipeline to run an automated performance test and see the result in Jenkins. To see all the scripts and data check this repo on Github

Basics about Performance Testing Tools

2019-12-31T05:00:00.001-08:00

I had an opportunity to give a speech about how we can automate performance testing in CI/CD pipeline in a conference held in Istanbul for the first time "Test automation and Digital QA Submit Istanbul, #TAS19". My subject was "Automated Performance Testing", the first part of the speech is to explain what is performance tests and the second part is to explain performance tools, and the last section is to explain what is automated performance testing and how to implement it to CI/CD.

In this post, I want to explain my thoughts about the basics of performance testing tools. Next posts will focus on automated performance testing and using K6 as a performance testing tool.

How to Evaluate Performance Testing Tools

Before diving into the performance testing tools and technologies we have in the industry, we should define the criteria for evaluation. Each tool has its own advantages or disadvantages and similarity between ancestors or others. When we start to use the tools and try to learn it, most of the time the learning curve is having a critical factor. It is fine since the learning is and using it efficiently is a huge impact for tools but after this period the performance of the tool during running it becomes more critical and having support from open-source community or consultancy companies is another very important factor. Let's list the factors for evaluating performance testing tools:

Criteria for Performance Testing Tools Evaluation:

Learning curve

Using a tool, in the beginning, can be difficult so a good tool should have well-defined documents and a better one ready project to run
Tools can have many features so running simple scripts doesn't mean that the tool is efficiently using for complex test scenarios. A good tool also support complex cases to handle real-world scenarios

Price

None of us wants to spend many if there is a free version of the tools. If the case is the money then you should go with the open-source tools but don't forget to search that the tool has a good community so that you get some support from there.
Some tools are created by the community so you can use without paying money, however if you want to get some enterprise-level support, some companies are giving this supports for the tools

Supports

Be sure that tool has a supportive community if it is open-source
Good company support for commercial tools

Maturity

Tools may have defects, so having a buggy tool integration in your environment may cause maintenance costs. Before investing in a tools check the known issues about the tools.

Languages supported for performance testing scripts

One can not have a profession in many languages so the supported language should include your favorite one
In general, performance scripts are easy in the beginning but if you want to cover complex scenarios with scalable test environment then the language may be a barrier to handle these situations

Performance of the tool itself

Testing tools require resources to create virtual users and requests. This resource may increase exponentially if you want to create lots of users for your stress tests so be sure that how the performance of the tools itself
Generally, the performance is evaluating by AB - Apache Benchmark tool. The response time and resource which is consuming by the tools are comparing with the result of the AB. To get the idea about the performance of some well-known performance testing tools, read this blog.

Integration to other necessary tools

How you can integrate it with databases, reporting tools, CI/CD and so on

Embed features

Separated load creation
Load distribution
Writing scenarios
Grouping the scenarios
Setting thresholds for automated performance tests

CLI supports

Supporting only GUI or CLI
Working headlessly
Dockerized or can be dockerized easily

Performance Testing Tools

We have a variety of tools in the industry, they change from open source to commercial and/or commercial support; newly started to 20+ years old performance testing tools. Some of the well-known performance tools, that I have experienced and I can suggest you, are as follows

Open-source

Jmeter - Apache project, v1.0 1998

Gatling - Enterprise support by Frontline

K6 - Enterprise support by LoadImpact

Locust

and many others

Commercial

LoadRunner by Microfocus (previously Mercury - HP)
Blazemeter - support multiple opensource tools
LoadUI Pro by Smartbear
and many others

Performance Testing

2019-12-16T01:31:00.000-08:00

In this post, I want to explain my thoughts about performance testing. Next posts will focus on automated performance testing and using K6 as a performance testing tool.

What is Performance Testing By Definition

Defining the performance of the system by testing the functionalities of the system in terms of the non-functionalities

Performance Testing is

A non-functional testing

Testing the functionalities of the system

Both black-box and white-box testing

Defining how the system works under any loads

Applied under different load levels

Applied under different load increments

Applied under different durations

What is Performance Testing in Terms of Process

Performance testing should be also executed in a process. Performance engineers should ensure that they collected requirements, created the correct amount of virtual users, created test scenarios and they can execute them. As a final, they should check the result and the system, if required after fixing, re-execution may be needed.

Performance testing should start with planning, in this stage, you should align with customer/user needs. Without understanding the requirements any test can not be evaluated. Two main parts of the testing are verification and validation. By verification, we are checking the requirements and by validation, we are checking these requirements are valid with the global expectation at least the targetted users. Therefore this stage is very important.

In the next steps, you should create performance test scripts and create test environment and in the execution stage, you can run test scripts against the requirements in the reserved/isolated performance testing environment.

In the monitoring stage, you should check that the execution is happening as expected by checking the system. The performance scripts create requests which are reached to the related part of the system and they create related data on the databases/caches. Which means that your scripts run as expected.

In the analyzing stage, you can check the result and if necessary you can fix some configurative, network-related or database-related problems. After that if necessary, you can get ready for the next iteration. These iterations should keep going until the desired level of confidence is gathered.

Types of Performance Testing

Types of performance testing depend on the strategy that applied during the test by means of the loads, the duration and increment of the load.

Load Test

The load test is to understand the behavior of the system under a specific load. It is performing under a specific amount of the load to check if the system gives the expected responses with keeping it stable for the long term

Load Test is

Basic form
Widely known-form
A specific amount of load
Checking the expected response time
Giving a result that users are facing

Stress Test

The stress test is to test the upper boundary of the system. When the load is increased to the level of the upper boundary, the stress test aims to get the response of the system.

Stress Test is

Testing the upper boundary of the system
Needing more resources
Needing more investigation for boundaries
Needing more attention
Prone to break the system, aka: soak, endurance testing
Preparation for promotion like black friday, 11.11

Soak - Endurance Test

Soak test is to test the upper boundary of the system for a prolonged time. Soak is good if you are not sure the expected amount of the user for some occasions.

Soak Test is

Testing the upper boundary of the system
Testing the system for a prolonged time
Needing more resources
Needing more investigation for boundaries
Needing more attention
Aims to break the system
Preparation for campaigns like black Friday, 11.11

Spike Test

Spike test is to test the behavior of the system when the amount of load is suddenly increased and decreased. It aims to find system failures when loads is changing to an unexpected amount.

Spike Test is

Testing the sudden increase and decrease of loads
Requiring more load generations
- depends on the tools (load the system, then spike it)
Good for some occasion like simulating
- push notifications for e-coms
- announcing critical news

Capacity Test

Capacity testing is aiming to find the maximum amount of the user that the system can handle. By this test we can find the how many user can use the system.

Capacity Test is

Testing the max user
Finding the behavior when more users join
Providing info about the loss of user data

Volume Test

The volume test is testing the database directly instead of the whole system. With the volume testing, we can find if there is any bottleneck with the database system.

Spike Test is

Testing the database
Testing the with a large amount of data

Getting the text of elements in Espresso

2019-12-06T00:47:00.000-08:00

Espresso is not designed to play with UI objects. When you are considering it in terms of capability for testing, it needs to be improved more. It is not as mature as Android SDK itself so you need to customize somethings to handle your requirements. Even you may need to use other tools like UIAutomator integrated into your test suite.

In this post, I want to show how we can handle getting the text of ViewInteraction view.

ViewInteraction view does not have .text function for getting the text of the object since it is designed to interact with the object. Another method which may solve the problem of the assertion is to use the .check for matching the text, as follows:

However, this method is not a good way to assert that a certain field text equals a value. For this purpose, we need to get the text of the element and assert it. To get the text of the ViewInteraction element, we need to cast it to TextView with getting the assignable form of it then extract the text that the view holder has. You can use the following function for this assertion:

Using it simple now, just call it whatever you want:

How to Set Shared Preferences in Espresso Test for Kotlin and Java

2019-05-03T05:33:00.000-07:00

I have experienced Espresso and needed to deep dive into Shared Preferences

just because it is one of the main parameters used in the application we developed. As a long search in the online sources but there are some pretty old documents for Espresso with Java and very few documents about Espresso with Kotlin. In this post, I want to share my experiences with setting Shared Preferences with Kotlin and Java and how you can use it in your test design. You can follow up the steps for your test project.

Shared Preferences is a way to store user data in local devices so it has been supported since the very early version of Android. Shared Preferences can be stored in the default file or custom file.

Using Default File for Shared Preferences

If your application uses the default file it should stores the shared data in the default file provide by Android as in the following path in the device:

/data/data/com.package.name/shared_prefs/com.package.name_preferences.xml

This is the source code for getDefaultSharedPreferencesName

For this case, you can set/clear/get shared preferences by the following method:

Using Customized File for Shared Preferences

The same way. your application can use a custom file to store shared preferences. This time you need to get the name of the .xml file for storing shared preferences. You can find the file in the same path with the default preference file. If the file name is `MyAppSharedPreferences`, the file path should be like the following:

/data/data/com.package.name/shared_prefs/MyAppSharedPreferences.xml

For this case, you can set/clear/get shared preferences by the following method:

When there is an update in shared preference/application data, we need to launch the application after this update. To do this, launchActivity should be set to false during initialization.

See the full code in Kotlin:
See the full code in Java:

(Micro) Service Testing with Postman - Newman - Docker

2019-01-03T04:53:00.001-08:00

Postman seems to become a defacto tool for service testing because the Postman is very user-friendly, easy-to-learn, all-in-one, lightweight and collaborating tool. Postman has been used for a long time but recently it has growing popularity because of a stable native application, collaboration feature after version 6.2, sharing of collections for team, interactive working with the team, mocks for isolated testing, environments for running the test for different test environments such as local, development, stage ... and many more features. For me, one of the biggest features is easy-to-use for everyone in a team so everyone in a team can use and update a postman collection easily. In this post, I want to explain how postman can be used efficiently.

Testing a Service and Writing Tests

With postman testing service is simple. Postman supports many methods like POST, GET, PUT, PATCH. Just select the correct method and hit the service URL you want to test. Postman also has everything that you need for complex requests headers, body, and parameters list. If you need something to prepare before sending the test request you can do it in the 'Pre-request Script'. If you need authorization you can add it 'Authorization' part which is an alternative to adding 'Authorization' parameter in the 'Header' section. If you want to check the not only the status code but also returning data you can add some assertions in the 'Tests' section. Postman supports Chai Assertion Library as BDD you can add self-descriptive assertion functions to your tests.

For complex scenario, you can write some javascript in the tab to simulate real behavior of your users. In the following scenario, a basic payment process is started with a post request and then the status of the request becomes `processing` but after that client needs to check the backend for each second until the status of the process become `completed`. For this scenario, in the `payment/payment-processing` request, until the status of this request is `processing` it calls itself recursively. If the status is not `processing` then it calls the next requests with another great feature of postman `setNextRequest` as `postman.setNextRequest("payment/payment-complete")` see the example:

Pre-Request Script For Creating Data Dynamically

The pre-request script is designed to run something before the test so we can use it creating/updating/deleting test data for comprehensive test cases. In this section, you can use the embed node packages like `atob` and `btoa` for base encryption or `cheerio` is a form JQuery library to get value from a web object or `CriptoJs` cryptographic library to encrypt data with AES, DES or SHA. You can check the full list of embed libraries from postman document.

In the following example, for RSA encryption with base64, I use an external web application. In the pre-requests tab of the request, `myData` is encrypted by sending a request to this web application. This is another important feature of the postman.

Running Postman Tests With Postman

You can run a collection with Postman collection runner easily by setting environment if necessary. Firs you can get my test collection which is written against to the blog project on GitHub. You can check the `service-test` section since it is a full project including every test that you need in your CI/CD. To run the tests, just click the play button next to the collection name, it opens the run panel then you click the `Run` button. See the following images:

Running Postman Tests With Newman

Newman is the CLI companion of postman so that you can run your tests with your environment by CLI. This is developed for integrating your test to CI/CD environment easily. In this example test data is created before the test run for a comprehensive test result. The full command is as follows:

You need node first, check this https://nodejs.org/en/download, then install the required packages, newman and newman-report-html:

npm install -g newman
npm install -g newman-reporter-html

to run with collection json:

newman run blog-sample-service.postman_collection.json \
 -e blog-local.postman_environment.json \
 --reporters cli,html \
 --reporter-html-template report-template.hbs

to run with collection url:

newman run https://www.getpostman.com/collections/ac3d0d9bbd8ae1bcfe5d \
 -e blog-local.postman_environment.json \
 --reporters cli,html \
 --reporter-html-template report-template.hbs

Running Postman Tests With Newman in Docker

For a better approach to run test on CI/CD pipeline is to use Docker so that we will not need to install any test related tools/languages/technologies to the client machine. This is becoming one of the industry standards.

Check the Dockerfile in the project. Basically, the base image is node installed Debian, we are installing the required node package which is newman and newman-reporter-html. Then we are running the test inside newman folder and the report will be created inside /newman folder. This Dockerfile gives us a newman entry point which can run it with newman commands. I have create Newman image in hub.docker, so we need to pull it first then we use it.

docker run \
 --network host \
 -v $PWD:/newman \
 gunesmes/newman-postman-html-report run https://www.getpostman.com/collections/ac3d0d9bbd8ae1bcfe5d  \
 -e blog-local.postman_environment.json \
 --reporters cli,html \
 --reporter-html-template report-template.hbs

We need to restore data during the test some data need to be ready. For each run, the database should be restored with the script provided in restore_database.sql. This file can be run by

bash restore_db.sh
bash run_service_test.sh

Get the Postman Collection and the project.

Scalable Tests for Responsive Web: Running Cucumber and Capybara with Tags in Dockers

2018-06-14T09:39:00.000-07:00

If you are using Capybara with Cucumber, tagging is a very efficient feature of the

Cucumber for manageable test cases. Tagging can be also used for managing test environments, like @test may mean that these tests run only in the test environment so set the environment as the test for me. Likely @live may mean that these tests can run on the live environment, assume that you are applying continuous testing. You can also give the scenarios for device-specific tags, like @ihphone6_v may say these tests are for iphone6 with the vertical mode.

Moreover, with tagging, you can also make an isolation strategy for running your test in parallel. Each parallel suits should have its own data like users, active records, address, credit card info, and ext. I have used the tagging for running tests in dockers. In this post, you can find some practical way of running Capybara with Cucumber in Dockers.

Creating Docker Image: Dockerfile

I am using Ruby version 2.3 so in Dockerfile getting the image of FROM Ruby:2.3 which is the first layer of the image comes as a Debian os. Then installing required libraries with the RUN command. Changing the working directory with WORKDIR which we will use this path by mounting the project folder into the running Docker. COPY Gemfile to WORKDIR so we can RUN bundle install to install required gems. The last thing, we need to install Chromedriver and Chrome by RUN commands. These steps executed by the Dockerfile, check it below:

Running Capybara in a Docker

Running the Capybara with my image is pretty simple if you run it without docker, just try other options which most suited for your case.

docker run --shm-size 256M --rm --name capybara_m \
 -v $PWD:/usr/src/app gunesmes/docker-capybara-chrome:latest bash \
 -c "cucumber features/G001_basket_product_checks.feature DRIVER=driver_desktop"

the meaning of the parameters in the command as follows:

run: running docker
--shm-size: Size of /dev/shm. Shared memory size, if you do not set it Chromedriver may be unreachable if there is no allocated memory.
--rm: Automatically remove the container when it exits so we can rerun if fails
--name: Assign a name to the container
-v: Bind mount a volume, we are mounting the local files, inside the container. $PWD:/usr/src/app means to mount the path present working directory to the path called /app inside the container
gunesmes/docker-capybara-chrome: name of the images which simulate the local machine works as Capybara machine. For the first time you run, it downloads the images from hub.docker.com, but for the later run, it will just download the updated container layers if there are.
:latest : version of the image, if you tagged your image when building it, you can use it.
. . .: and the rest of commands are for the Cucumber specific. You can see them in Cucumber docs.

Visual Run for iPhone6

For debugging purposes you may need to run the test visually to see if everything happens as expected. For this purpose set the visibility of the driver as visible by selecting the one in the list of env.rb file. Therefore the following command simply runs the test via iPhone 6, vertical mode visible. You can also see the video for this run.

cucumber features/G001_basket_product_checks.feature DRIVER=driver_mobile_iphone6_v_visible

Docker run for a single tag with a single platform

You can run a feature for a single platform by defining the DRIVER option with the desired one. In the following run command we can run one feature file for the driver running on the desktop.


docker run --shm-size 256M --rm --name capybara_m \
 -v $PWD:/usr/src/app gunesmes/docker-capybara-chrome:latest bash \
 -c "cucumber features/G001_basket_product_checks.feature DRIVER=driver_desktop"

Docker run for parallel execution of smoke test

bash run.sh

check the reports inside the ./html_report

Docker run for parallel execution of all tags with all platforms

bash run_full.sh

When we run the run_smoke.sh, there is only two parallel run executed against the desktop version of chrome and iPhone 6 version of chrome with the vertical mode. Mobile for better experiences we both set the dimension of the mobile devices and the user-agent for mobile device operating system. However, there is still a technical problem that we don't run the test in real devices. Check the browser object in the env.rb. You can run the run_full.sh in a dedicated environment, it just requires more resources than an ordinary machine.

In the end, it creates JSON report files in the ./report folder so that you can create nice HTML reports with cucumber-JVM report plugin for Jenkins. It includes screenshots on error and the logs of the browser.

Check the result of the log, you can the benefit of the parallel run. Each test takes approximately 1 minute when we run the run_full.sh, there are 7 parallel runs but all test takes approximately 1minute 20seconds.

Isolated - Scalable Performance Testing: Locust in Dockers

2018-05-31T16:45:00.000-07:00

I have shared some posts how to run Locust in Local or in Cloud, as slave or master. At this time I want to share how you can run it in a Docker. To fully get the benefits of Locust I am using it with Python 3 so I created a Docker file and images upload to the Docker Hub, and the project is on the GitHub.

The Dockerfile has the minimum available requirements but we also have a new file called `requirements.txt` which you can add required python libraries to install inside the container by pip install lib. The Dockerfile is:

When you got the docker-locust image, you can now run your script. At the end of the Dockerfile you can see that ENTRYPOINT [ "/usr/local/bin/locust" ] this enable us to use the image as service which means that you can directly call it same as using Locust installed locally. See the run command below:

Running Locust in a Docker

Running the Locust with my image is pretty simple if you used without docker, just try other options which most suited for your case.

docker run --rm --name locust -v $PWD:/locust gunesmes/docker-locust -f /locust/run.py --host=http://www.github.com --num-request=100 --clients=10 --hatch-rate=1 --only-summary --no-web

the meaning of the parameters in the command as follows:

run: running docker

--rm: Automatically remove the container when it exits so we can rerun if fails

--name: Assign a name to the container

-v: Bind mount a volume, we are mounting the local files, run.py, inside the container. $PWD:/locust means to mount the path present working directory to the path called /local inside the container

gunesmes/docker-locust: name of the images which works as Locust service so it is a synonym of the locust . For the first time you run it downloads the images from hub.docker.com, but later it will just download the update container layer if there is.

. . .: and the rest of commands are for the locust specific. You can see them at the end of the post.

See what is happening when you run the command for the first time:

List of Locust command, note that Locust will remove the --num-request in favour of stopping it by a timer parameter.

The little but powerfull run.py is just collecting the link inside the hostname home page, and starting to attack by Locust when it is ready. See the run.py: