Blue-Bag - Information Systems

Drupal - My view has a null module dependency

George Boobyer — Thu, 01 Jul 2021 16:16:12 +0000

We often get asked to troubleshoot issues on Drupal sites and recently, twice in succession on different Drupal 8 sites, we came across an issue where a configuration file contained the word null in its list of module dependencies.

dependencies:
  module:
    - null

This was causing a number of issues and is clearly wrong. When trying to import configuration for example you are faced with a problem because Drupal thinks the configuration relies on a module called null which clearly doesn't, and never did, exist.

So just create a module called 'NULL' - Problem solved :p

There actually is a Drupal contributed module 'null' that is provided specifically to overcome this issue - but it effectively null routes the error by providing a module called null to match the dependency - it addresses the symptoms not the cause - https://www.drupal.org/project/null - if you are in a desperate hurry to move on with your life then that might be for you - if you want to solve the cause read on.

If this does happen to you then you will have problems importing configuration and in our case the view whose configuration file was affected also raised an error when being saved.

Notice: Undefined index: table in Drupal\views\Plugin\views\relationship\RelationshipPluginBase->calculateDependencies()
(line 179 of core/modules/views/src/Plugin/views/relationship/RelationshipPluginBase.php).

Let's investigate

I thought it would be a good opportunity to write up the investigation as I didn't find a lot of similar issue when Googling and I am sure I'll come back here at somepoint myself.

This is relevant to Drupal 8 and Drupal 9

In the scenario that follows the site had a custom module that is used to attach Promotions to line items. The module creates a number of tables for this purpose and declares a schema to describe the tables and their fields. These tables and some of their fields are used in the affected view.

So what is going on! - Why is the configuration writing null as a dependency?

The view's configuration file contained the following excerpt:

uuid: 12345678-1234-5678-ab9ba-9876d54cba32
langcode: en
status: true
dependencies:
  module:
    - null
    - csv_serialization
    - mycustom_schema
    - rest
    - serialization
    - user
    - views_autocomplete_filters
    - views_data_export
    - views_date_format_sql

Custom schema tables

The tables in this case are:

A custom line item table
A table to contain data about promotions and
A join table to link them (I have simplified this for illustration)

The view surfaces some of the fields from the Line Item and Promotion tables and it all works ok in the view - but when you save the view you get:

+---------------+                     +--------------+
|               |                     |              |
|  Line Item    |                     |  Promotion   |
|               |                     |              |
+-------+-------+                     +-------+------+
        |                                     |
        |                                     |
        |    +-----------------------+        |
        |    |                       |        |
        +--->|   LineItemPromotion   <--------+
             |                       |
             +-----------------------+

Notice: Undefined index: table in Drupal\views\Plugin\views\relationship\RelationshipPluginBase->calculateDependencies()
(line 179 of core/modules/views/src/Plugin/views/relationship/RelationshipPluginBase.php).

But the view works.

Roll up your sleeves

So how do we troubleshoot this and find out what is causing the issues:

You get an error when you save the view,
The config for the view has null in the name of a dependency and
You can't import the configuration

The first thing to check is if the schema of the data model is implemented correctly in the modules install file in

hook_schema()

This is where you define the schema so the tables get created when the module is installed. When investigating this it became apparent that the tables were modified in subsequent and successive updates (adding new fields and indexes etc)

hook_update_N()

As it happens, although the schema was updated by these updates, the module schema itself was not updated - this was not the source of our problems, but would be a problem when the module was installed in the future.

see Updating Database Schema and/or Data in Drupal 8 (www.drupal.org/docs/drupal-apis/update-api/updating-database-schema-andor-data-in-drupal-8)

You should always update the schema with such updates.

Plugin Dependencies

But fixing that does not solve our specific issue. Furthermore, our module is present in the list of dependencies for the view and the view works (although it errors when saved and we can't import the configuration). So we need to see why Views can't work out the dependency and places a null in the config.

To do this we need to understand how Views determines the dependencies and see what the actual source of the error is - maybe not our module after all?

When the configuration for a view is written, Views calculates the dependencies of fields and other things (plugins) used in the view. It does this in RelationshipPluginBase:

web/core/modules/views/src/Plugin/views/relationship/RelationshipPluginBase.php
abstract class RelationshipPluginBase extends HandlerBase {

  .... other stuff removed for readability ....
172:   /**
173:    * {@inheritdoc}
174:    */
175:   public function calculateDependencies() { 
176:     $dependencies = parent::calculateDependencies();
177:     // Add the provider of the relationship's base table to the dependencies.
178:     $table_data = $this->getViewsData()->get($this->definition['base']);
179:     $dependencies['module'][] = $table_data['table']['provider'];
180:     return $dependencies;
181:   }
182:
183:}
184:
185:/**
186: * @}
187: */

This is called for all of the plugins (fields in our case) required by the view and this in turn calls getViewData() to find out which module each plugin comes from, to build up a list of dependencies.

When we come to line 178 we go off to web/core/modules/views/src/ViewsData.php which looks at all modules that implement hook_views_data() to see which is the originator of the plugin.

235:   protected function getData() {
236:     $this->fullyLoaded = TRUE;
237: 
238:     if ($data = $this->cacheGet($this->baseCid)) {
238:       return $data->data;
239:     }
240:     else {
241:       $modules = $this->moduleHandler->getImplementations('views_data');
242:       $data = [];
243:       foreach ($modules as $module) {
244:         $views_data = $this->moduleHandler->invoke($module, 'views_data');
245:         // Set the provider key for each base table.
246:         foreach ($views_data as &$table) {
247:           if (isset($table['table']) && !isset($table['table']['provider'])) {
248:             $table['table']['provider'] = $module;
249:           }
250:         }
251:         $data = NestedArray::mergeDeep($data, $views_data);
252:       }
253:       $this->moduleHandler->alter('views_data', $data);
254: 
255:      $this->processEntityTypes($data);
256: 
257:       // Keep a record with all data.
258:       $this->cacheSet($this->baseCid, $data);
259: 
260:       return $data;
261:     }
262:   }

From line 244 on it gets a list of tables for all modules that implement hook_views_data and looks for the table name from the plugin.

If none is found it returns an empty $data and this causes the error:

Undefined index: table

when the RelationshipPluginBase (line 179) is called resulting in 'null' being returned as a dependency.

So in our case we find that when looking for the module that implements the table (mycustom_schema_line_item_promo) we are returned 'null' since it is not found.

The other tables and more importantly the fields they display in the view results are fine.

So we need to investigate the implementation of the hook_views_data in the custom module.

Declare which tables you own

What we find is that the tables (mycustom_schema_line_item and mycustom_schema_promo) whose fields are in the view (and correctly return the module as the dependency) are defined in the module; as are their relationship with the join table. But, the join table itself is not declared in the hook_views_data() and therefore views can't figure out which module it is dependent on.

The following extracts from this hook implementation show how they are declared:

For brevity I have omitted the declarations of all of the fields for the tables (and other custom tables).

// The line item custom table
  $data['mycustom_schema_line_item']['table']['group'] = t('Custom Line Item');

  $data['mycustom_schema_line_item']['table']['base'] = [
    'title' => t('Custom Line Items'),
    'help' => t('Custom Line Items provided by the Custom module.'),
  ];

  // ID.
  $data['mycustom_schema_line_item']['id'] = [
    'title' => t('Line Item ID'),
    'help' => t('Line item unique identifier.'),
    'field' => [
      'id' => 'numeric',
    ],
    'sort' => [
      'id' => 'standard',
    ],
    'filter' => [
      'id' => 'numeric',
    ],
  ];

  // The field to link in the promo.
  $data['mycustom_schema_line_item']['promo'] = [
    'title' => t('Promotion'),
    'help' => t('The names of the promotions applied to this order.'),
    'relationship' => [
      'base' => 'mycustom_schema_line_item_promo',
      'base field' => 'line_item_id',
      'field' => 'id',
      'id' => 'standard',
      'label' => t('Promotion'),
    ],
  ];


  $data['mycustom_schema_promo']['table']['group'] = t('Custom Promotion');

  $data['mycustom_schema_promo']['name'] = [
    'title' => t('Promotion'),
    'help' => t('The name of the promotion reference.'),
    'field' => [
      'id' => 'standard',
    ],
    'sort' => [
      'id' => 'standard',
    ],
    'filter' => [
      'id' => 'string',
    ],
  ];

  // the join details to link the join table to the promo table
  $data['mycustom_schema_promo']['table']['join'] = [
    'mycustom_line_item_promo' => [
      'left_field' => 'promo_id',
      'field' => 'id',
    ],
  ];

return $data;

So although the relationships are defined here (interestingly in two different ways - but let's not get distracted) there is no definition for the link table (mycustom_line_item_promo) itself.

So when Views calculates the dependencies, it can find the details for the Line item table (mycustom_schema_line_item) and Promotions table (mycustom_schema_promo) and their fields but not for the join table (mycustom_schema_line_item_promo) so can't determine which module it is dependent upon.

We need to add some metadata for that table to the hook_views_data().

 
....  
  
  // in our case all we need to add some table meta data so views can see we 'own' the table
  $data['mycustom_schema_line_item_promo']['table']['group'] = t('Custom Promotion');
  
  $data['mycustom_schema_line_item_promo']['table']['base'] = [
    'title' => t('Custom Line Items - Link to promotions'),
    'help' => t('Custom Line Item promotions provided by the Custom module.'),
  ];
  // why not the fields too? Maybe because we don't want them to be included in views results?
  $data['mycustom_line_item_promo']['id'] = [
    'title' => t('Line Item Promo ID'),
    'help' => t('Line item promo unique identifier.'),
    'field' => [
      'id' => 'numeric',
    ],
    'sort' => [
      'id' => 'standard',
    ],
    'filter' => [
      'id' => 'numeric',
    ],
  ];

Once this is added and caches are flushed (the dependency data are cached in the ViewsData getData() so this is important) the view can be saved without error, the config no longer contains a null dependency (it already had the dependency for our module obtained from the other tables that were declared) and we can import the config.

Hopefully this will point others in the right direction if they face the same issue. Of course it might not be Views at the heart of your issue - if you have faced this issue let me know in the comments.

Category:

Drupal

Blog image:

BOM disposal

George Boobyer — Tue, 24 May 2016 08:41:56 +0000

Occassionally when you get files from someone else or use a new text editor you encounter an issue with hidden characters or incorrect encoding that can be a pain to track down.

A common one of these that I haven't encountered for ages, (since we moved from Windows to OSX for development) is the Byte Order Mark (BOM).

This is a simple marker that indicates the encoding order of UTF-8 files and should in itself be no problem. It does however cause an issue when the file is included within other files (by PHP as *.inc files for example) such as in template files.

This can result in blank lines that are hard to track down or problems with page headers.

You may also find that the BOM causes problems for an ordinary PHP page. When sending custom HTTP headers the code to set the header must be called before output begins. A BOM at the start of the file causes the page to begin output before the header command is interpreted, and may lead to error messages and other problems in the displayed page.
https://www.w3.org/International/questions/qa-byte-order-mark

It was exactly that issue that I encountered the other day and being such a blast from the past - I thought I would make this quick post about the issue and some ways of tracking it down and fixing it.

So, first off what issue does it cause. Well in my case and in past cases too it caused Drupal to complain and create a circular redirect on admin pages.

This is illustrated by errors such as the 'headers already being sent’ error (https://www.drupal.org/node/1424).

In fact it was that error that jogged my memory on the subject as I hadn't seen it for a long time.

The cause was an old module template file that I hadn't done any work on for a few years since moving from Windows development environment.

So in this case I was thrown a bone as that was obviously the latest change to the site.

In other cases however it can be tricky to track down - so lets look at what it is and how to locate and remove it.

You can search your files for BOMs (limiting the search to the first line, as the character sequence may appear elsewhere and it is only the first characters of the first line that we are interested in).

grep -rl '\xEF\xBB\xBF' .

on MACOSX you need to prepend a dollar sign:

grep -rlI $'\xEF\xBB\xBF' .

Note the full stop is required to indicate the files in the current directory.

We could remove this en masse with a script - but this may cause issues if the file is supposed to have them:

sed 's/\xEF\xBB\xBF//' < inputfile > outputfile

again on MAC oSX (note the $ )

sed $'s/\xEF\xBB\xBF//' < inputfile > outputfile

Note that these will remove the first instance (if you wanted to remove every instance - not adviseable - but as 'sed' note - use the global modifier 'g').

However unless you have loads of them it is likely to be only a single file that is causing an issue - so lets open that and see what it looks like.

vim file.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

Although the BOM is the first sequence of charatcetrs in the file - the BOM is not displayed by defauklt in most text editors. This makes them hard to locate/assess.

But you can open the file in VIM in binary mode::

vim -b file.html

<feff><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

Note the <feff> char sequence.

When looking at the file in VIM in normal text mode you can check if there is a BOM:

:setlocal bomb? 
bomb

And get it to save without the BOM

:setlocal nobomb 
:w

Most text editors allow you to set a a global option to not set the BOM.

See the attached Asciiinema of a terminal session showing these steps (or go here https://asciinema.org/a/46517).

props:

http://stackoverflow.com/a/12220257

http://vim.1045645.n5.nabble.com/How-to-display-and-remove-BOM-in-utf-8-encoded-file-td4681708.html

http://stackoverflow.com/a/29490814

Category:

Drupal

Blog image:

Stock for Drupal 8 commerce - Technical overview & use cases

Guy Schneerson — Wed, 11 May 2016 11:52:05 +0000

The business requirements

Drawing on the experience of maintaining the Drupal 7 stock module and working with a number of ERPs over the years we have come up with a completely new architecture for Drupal 8 commerce stock module.

The following are the 4 main challenges that I encountered with the Drupal 7 module that needed addressing:

Performance issues including deadlocks
The need for a mature transactional based system
Location based stock
Integrate with external ERP systems.

At the same time it was important to me to make sure the module is suitable for the majority of the stock module users that were happy with the simple approach of the Drupal 7 version.
In so doing we have made it easy for others to extend.

The result is a proposed architecture that addresses all these and has been presented over the last year through the drupal.org commerce stock issue queue

The Service architecture

The StockAvailabilityChecker - This implements the core commerce AvailabilityCheckerInterface and, once this interface is implemented in commerce, it will provide the stock checking functionality.
This service doesn’t do much in itself as it uses the StockManager to do the work.

The StockManager - This is a service collector that collects services of type StockServiceInterface. It also holds a configuration class that tells it when to use this service.
Currently the configuration will allow for selecting one active service but at a later stage this will be extended to work at a more granular level (probably by product type).

The Stock service - The module comes with two services: The first is an "Always in stock" and the second is the "Transactional Stock" currently called "Local Stock" (i.e. stored in the drupal DB).
Each stock service needs to implement 3 classes based on the following interfaces: StockCheckInterface, StockUpdateInterface & StockConfigurationInterface.
The Configuration determines the relevant location(s). The checker does the availability checking and the Update the updating.

The Transactional Stock Service

This is designed to provide full audit capabilities on current stock levels, far better tracking and reporting and improved performance across multiple locations (warehouses, stores, ...)
The idea is that each stock movement (sold products, new stock ..) will result in a transaction record. This record will hold a quantity along side other information.
These records can be summed up by product variation to get the overall stock level.
The advantage of this system is that by using only selects and inserts we get a highly efficient db system that can not result in deadlocks (a common D7 stock issue). The potential problem is that large number of transactions can be expensive to sum. The solution for this problem is a cron job that will periodically sum existing transactions and update a master record with the totaled quantity alongside the last transaction id. This introduces an update but as this is handled by a single process (cron) we are not introducing a potential deadlock so the result should provide performance and scalability.

The state of things

The current implementation of the Drupal 8 module available on drupal.org provides a complete service architecture and it's the Stock service itself that is the focus of on going development.
The transactional stock service implements an API capable of creating transactions and running availability checks, so once we have an interface (currently only a dev one is available) & commerce implements the availably Checker we should have a working system.
However the following need implementing:

The cron job for updating the location stock
UI for creating and managing stock locations & transactions. Currently a development sub module for calling API functions is included but will be removed when the first beta is realised.
Transaction Metadata holding information for each transaction - In progress to be finished shortly.
Unit Cost - The transaction record includes a unit_cost field for for FIFO and LIFO reporting.

Use cases

The plan is to sub-class the current service "local Stock" into two services that will be almost identical but serve two use cases.

Transactional stock - A full transactional service or a 'non watered down version' of "local Stock"
Simple stock - This will use the "local stock" functionality but delete transactions after they are totalled to avoid holding a large dataset when one is not needed. It is likely to include a more simplified interface and possibly be limited to a single location.

Other use cases for the module are:

External ERP integration - New services can be developed for specific ERP systems I am hoping to develop the Brightpearl one see https://www.drupal.org/project/commerce_brightpearl in time for a full release.
Other use cases and customisation - The combination of a the service architecture + the fact that Drupal 8 is OOP will allow people to derive their own services from existing ones as well as developing a complete solution to be plugged into the StockManager.

How can I help?

If you have an interest in inventory management or just want to help make Drupal and Drupal Commerce better do get involved with the issue queue.

for more details see https://www.drupal.org/project/commerce_stock and Stock for D8: Architecture & high level functionality in the drupal.org issue Q

Category:

Drupal

e-commerce

Drupal 8 Release

George Boobyer — Tue, 17 Nov 2015 15:38:18 +0000

On the 19th November 2015 a new version of the Drupal content management system (CMS) will be released - Drupal 8.

This new version has been over four years in development and has seen contributions from over three thousand people. It contains well over 200 new features, uses object orientated code and is probably the most significant release to date.

There will be over 200 release parties around the world on the 19th; a series of events that will bring the Drupal community together to mark this new release and discuss the opportunities and implications of the most powerful versuin of Drupal yet. See a map of all of the event - http://www.drupical.com/

Drupal Somerset (@drupalsomerset) is planning on holding it's own release party in Glastonbury to join in this global celebration.

If you are in the area and would like to attend then let us know @drupalsomerset on twitter or contact us through this website's contact form.

Indigo-herbs has kindly offered to host the event at their new development offices in Glastonbury:

Indigo Herbs
Unit 8
Adlams Central Park
Wirrall Park Road
GLASTONBURY
BA6 9XE

The plan is to meet at 19:00. There will be refreshments and the opportunity for a range of discussions around the following topics.

What’s new in D8
Getting ready for Drupal 8
Trying Drupal 8 out: Getting Drupal 8 set up on a VM
When should I think about using D8 in my new projects

These won't be formal presentations - but more an opportunity to discuss these topics with a range of people; of which some have already used Drupal 8 and others who have not yet had the chance.

Everyone is welcome to attend.

Category:

Drupal

Blog image:

Compiling Varnish modules

George Boobyer — Mon, 09 Nov 2015 13:57:51 +0000

Varnish is a powerful tool and although simple to get running initially can have a steep learning curve beyond the initial install.

In this post I look at compiling a Varnish module and some issues you may face and how to overcome them.

When implementing Varnish, sometimes you want to cache pages for a 'test' site or a site that isn't open to the public yet.

Typically you would use basic authentication on the Apache server and have Varnish not cache pages that require authentication.

But if you do want Varnish to cache pages for a 'closed' site then you have to move authentication to Varnish.

There are two modules that support this -

basic authentication: which works like basic authentication in Apache and can use the same htpasswd files - https://www.varnish-cache.org/vmod/basicauth
authentication - that uses hases of user:passwrd in the vcl itself. - https://www.varnish-cache.org/vmod/authentication

I recently evaluated the basic authentication module: https://www.varnish-cache.org/vmod/basicauth - source: http://git.gnu.org.ua/cgit/vmod-basicauth.git/

The first problem I faced was the scant documentation about how to use Varnish modules.

tl;dr - Get the source for the module and Varnish, build Varnish and then build the module against the Varnish source.

I came across a detailed explanation of how to do this here : http://stackoverflow.com/questions/28163273/varnish-4-basic-authentication

In order to be able to use a Varnish module you need clone the module from source and compile it against the Varnish source.

To do this you have to get the Varnish source (and many dependencies) and build it (although you don't have to install it if you have already done so via your package manager).

$agt-get source varnish

To build Varnish from source follow the instructions here: https://www.varnish-cache.org/docs/trunk/installation/install.html and make sure you have the dependencies listed for your platform.

However the issue I faced was that the Varnish source would not compile.

This was due to the automake functionality not recursing into subdirectories.

warning: source file <some source file> is in a subdirectory,
but option 'subdir-objects' is disabled

To solve this you have to add the subdir-options option to the automake in line 18 of the configure.ac:

$vi /usr/src/varnish-4.0.3/configure.ac

AM_INIT_AUTOMAKE([1.11 foreign color-tests parallel-tests subdir-options])

However with this option there is an issue (on Debian at least) whereby variable expansion fails in the make files in the bin directory for the Varnish programs (varnishadm and varnishhist). (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=402727).

To solve this you have to edit the Makefile.am files in the varnishadm and varnishhist bin folders:

$vi bin/varnishadm/Makefile.am

varnishadm_SOURCES = \
	varnishadm.c \
	$(top_srcdir)/lib/libvarnish/vas.c \
	$(top_srcdir)/lib/libvarnish/vsa.c \
	$(top_srcdir)/lib/libvarnish/vtcp.c \
	$(top_srcdir)/lib/libvarnish/vss.c

varnishadm_SOURCES = \
	varnishadm.c \
	../../lib/libvarnish/vas.c \
	../../lib/libvarnish/vsa.c \
	../../lib/libvarnish/vtcp.c \
	../../lib/libvarnish/vss.c

Otherwise when you run the make command, the build will fail and you will have subfolders in the bin/varnishadm folder called $(top_srcdir).

This only appears to affect the SOURCES element. - see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=402727

Once that is done - you can build the varnish source and then compile and install your module as outlined in http://stackoverflow.com/questions/28163273/varnish-4-basic-authentication

Category:

varnish

Blog image:

Ansible filters for taming lists (part 1)

George Boobyer — Fri, 27 Mar 2015 16:08:47 +0000

When running through a series of tasks in Ansible it is common to want to have data available in a few formats - sometimes as a simple list and sometimes as a dictionary. What we don't really want to do is to have the same data in two or more separate lists (that would be hard to maintain).
So say I have a task that updates a series of images on the remote host and I need the data in the format -

{ src: filename, type: filetype}

But in another task I want a list of the types we updated. I don't want to have to manage two lists and I don't want to repeat an action if the types are duplicated across a number of file instances.
Ideally we would like to be able to get from the list a list of unique values from one of the elements - and that is where Jinja filters can help.
Firstly I am bound to say that Ansible is not a programming language, so we should always think of the simplest way to do this and not have loads of Jinja processing within a task.
Also in some cases it is worth considering creating/using a plugin to get the processing out of your tasks and templates (and that is the subject of part 2).
So back to the task - Lets consider a fictitious scenario. I have a site that has pictures of fruit and you can download archives of all of the images of fruit combined by type. A routine on the server builds archives for the fruit types that contain all of the images of fruit of that type. So I can request a download of a zip containing all citrus fruit images or all non citrus fruit images. These archives are cached so they don't have to be built more than once - but the cached files need to be dropped if more fruit images are added.
Occasionally I want to upload more fruity pictures and when I do that I want to drop the archives for the type (so they will get rebuilt when next requested). But I don't want to maintain two vars; one for the list of images and one for the list of types of archive to drop. What I want is one list of fruit images that lists the name of the files and the type of the fruit.
So I could easily use a with_items iterator for both a copy task (to put up the images) and the drop archive task (file state=absent), but that would repeat the second task necessarily.
So I create a dictionary var for the fruit images:

images_to_update:
      - { name: "oranges-001.jpg", type: "Citrus"}
      - { name: "oranges-002.jpg", type: "Citrus"}
      - { name: "apples-001.jpg", type: "Non-citrus"}
      - { name: "bananas-001.jpg", type: "Non-citrus"}
      - { name: "grapes-001.jpg", type: "Non-citrus"}
      - { name: "grapefruit-001.jpg", type: "Citrus"}
      - { name: "pears-001.jpg", type: "Non-citrus"}

and then create one task to use a with_items to copy each of these up to the server.
Then I use a filter to set a fact (a new variable that persists only during the scope of the play) to create a unique list of the type values so that I can drop the relevant archives.
This uses the filter:

fruit_archives_to_drop: "{{images_to_update|map(attribute='type')|list|unique }}"

This says take the dictionary structure (images_to_update) and create a list from the 'type' attribute and further make it unique - Cool huh! You can then use this generated list to iterate without repetition.

"['Citrus', 'Non-citrus']"

See a working demo of this in the gist below.
In the next part I will show how we can use this sort of thing to get a list of unique IPs from your inventory (hostvars) to add to a firewall whitelist or similar in partnership with a plug-in.

Category:

Ansible

YAML

jinja2

Blog image:

YAML Death by non-breaking spaces

George Boobyer — Thu, 19 Mar 2015 14:49:14 +0000

Just when you are in the flow of it - something comes up that spoils your mood.
So that I don't forget and hopefully as a note for anyone else tripped up by this.
Today a long play that I have been working on for some time suddenly started to fail with the message:

Traceback (most recent call last):
  File "/usr/local/Cellar/ansible/1.8.4_1/libexec/bin/ansible-playbook", line 326, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/usr/local/Cellar/ansible/1.8.4_1/libexec/bin/ansible-playbook", line 266, in main
    pb.run()
  File "/usr/local/Cellar/ansible/1.8.4_1/libexec/lib/python2.7/site-packages/ansible/playbook/__init__.py", line 313, in run
    play = Play(self, play_ds, play_basedir, vault_password=self.vault_password)
  File "/usr/local/Cellar/ansible/1.8.4_1/libexec/lib/python2.7/site-packages/ansible/playbook/play.py", line 165, in __init__
    self._tasks      = self._load_tasks(self._ds.get('tasks', []), load_vars)
  File "/usr/local/Cellar/ansible/1.8.4_1/libexec/lib/python2.7/site-packages/ansible/playbook/play.py", line 636, in _load_tasks
    data = utils.parse_yaml_from_file(include_filename, vault_password=self.vault_password)
  File "/usr/local/Cellar/ansible/1.8.4_1/libexec/lib/python2.7/site-packages/ansible/utils/__init__.py", line 786, in parse_yaml_from_file
    process_yaml_error(exc, data, path, show_content)
  File "/usr/local/Cellar/ansible/1.8.4_1/libexec/lib/python2.7/site-packages/ansible/utils/__init__.py", line 719, in process_yaml_error
    %s""" % (path, mark.line + 1, mark.column + 1, before_probline, probline, arrow)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)

This horrible error message was quite hard to debug - not because it isn't obvious as to what has happened (the last line says it all) - a stray extended character - dopey key press - has crept into one of my yaml files - but because the character was hard to track down.
It turned out to be a non-breaking space entered by mistake by option-space (a key press that normally brings up iTerm in visor mode - so not a mistake I make very often).
By turning on 'show invisibles' it still may not show up as it looks like a space. So hunting these things down can be hard.
So in my case a simple non breaking space (U+00A0) tripped me up.
But by re-tracing your steps and checking the encoding of your recently changed files (commit often!) hopefully you will spot such things before heading off on a goose chase.
Anyhow - putting it up here so I remember and hopefully to help some other poor passer by!
Here is a good resource for decoding such issues - http://rishida.net/tools/conversion/

Category:

Ansible

YAML

Blog image:

Using Ansible to protect against 'Connection attempts using mod_proxy'

George Boobyer — Mon, 13 Oct 2014 09:43:26 +0000

I regularly encounter issues that arise from reviews of server security, log reviews etc that provide good examples of how Ansible can be used to respond to an issue.
Once such example is making sure that all of your servers are protected from being used as a Proxy.
In our case we use Ansible to provision and monitor the configuration of our servers and we have a set of playbooks that secure our servers and set up security applications and firewalls to ensure they are not vulnerable.
A common set up is to secure you servers with IPTables and Fail2ban and use Logwatch to monitor your logs for common signs of unwanted activity. You should monitor these reports regularly and respond to any unusual activity; in most cases these should already be protected against.
But if you are using a utility like Logwatch to keep an eye on activity on your servers you may occasionally see an entry such as

   Connection attempts using mod_proxy:
   1.164.40.29 -> mx3.mail2000.com.tw:25: 1 Time(s)

This indicates that someone is scanning your server to see if they are able to use it as a Proxy to attach to a mail server or some other service. This is definitely not wanted and it is worth checking that your are not vulnerable to this exploit.
By default, Apache has ProxyRequests set to Off and doesn't by default install with the proxy modules enabled - but it is worthwhile checking to ensure that that is the case.
To do so by hand would involve something like the following:

Visit every server and check that the relevant modules are disabled.
Limit use of the CONNECT verb to ensure that such connections are denied.

That can be time consuming and you would need to do it regularly to ensure that the module is not enabled later on.

So using Ansible we can do the following:

Provide a list of modules we always want to see disabled on a server and create a task that will ensure they are disabled.
Provide a default configuration file on the server to limit use of the CONNECT verb and create a task that puts that in the correct place and restarts the web server. Note that the examples will require adaptation for some distros (i.e. Centos) and will require a handler for the Apache restart. In most cases these would be additions to your existuing playbook for controlling your web servers.

Doing it this way :

Documents the change to your build spec
Applies the measure to all servers.
Maintains the state of the measure
Enables you to extend how you respond to the issue

So we create a variables list (apache_mods_disabled) that contains all of the modules we want to be sure are disabled.

---
- hosts: webservers
vars:
apache_mods_disabled:
- proxy
- proxy_ftp
- proxy_http
- proxy_connect

We then use this list in a task that disables the modules if they are enabled (and does nothing if they are already disabled).

 - name: Apache | Disable Apache Mods
   apache2_module: state=absent name={{ item }}
   with_items: apache_mods_disabled
   notify: restart apache

We also create a small configuration file to limit the Connect verb and place that in the conf.d folder (read in by Apache on a restart)

# {{ ansible_managed }}
<Location />
<Limit CONNECT>
Order deny,allow
Deny from all
</Limit>
</Location>

and place that on the server using the template module.

 - name: Apache | Limit Connect verb
   template: src=apache_conf_connect.j2 dest=/etc/apache2/conf.d/proxy_connect.conf
   notify: restart apache

Each of these tasks will notify a handler to restart Apache if needed.
Typically you would simple add the variables and tasks to your existing playbooks.
You can the be confident that the protection measure is in place on all of your web servers and that state is maintained.
See a full Gist here:

Category:

Ansible

Apache

Security

The Team

bb-admin — Tue, 23 Sep 2014 10:44:40 +0000

The Team

In 2000 Blue-Bag was created by George Boobyer and Guy Schneerson to pool their experience and expertise.

Both George and Guy had been working for many years developing Information systems for a wide variety of clients and platforms. In 2010 we took the decision to move to open source technologies and moved away from proprietary and bespoke Windows based software and IIS/ .Net based web site development. Since then we have specialised in providing Drupal solutions on Linux hosting. This has enabled us to provide extremely competitive solutions that fit our customer's needs and don't tie them down to us as a provider - Our clients stay with us because they want to not because they have to!

Although we are based in the UK we have undertaken development work for clients across the globe from the USA to Russia.

Our Clients

bb-admin — Tue, 23 Sep 2014 10:43:53 +0000

Blue-Bag - Information Systems

Drupal - My view has a null module dependency

So just create a module called 'NULL' - Problem solved :p

Let's investigate

Custom schema tables

Roll up your sleeves

Plugin Dependencies

Declare which tables you own

BOM disposal

Stock for Drupal 8 commerce - Technical overview & use cases

The business requirements

The Service architecture

The Transactional Stock Service

The state of things

Use cases

How can I help?

Drupal 8 Release

Compiling Varnish modules

Ansible filters for taming lists (part 1)

YAML Death by non-breaking spaces

Using Ansible to protect against 'Connection attempts using mod_proxy'

The Team

The Team

Our Clients

trusted by