Middleware Snippets

From WildFly to WebLogic

2020-05-21T11:53:00.007+02:00

In this post we are going to use the foreign server or the messaging bridge to access third party JMS resources, such as activemq.

Set-Up WildFly

First download a WildFly distibution. In this example, we are going to use 19.1.0.Final.

To install WildFly simply extract the download. As we are going to use messaging, we are going to use the standalone-full.xml configuration.

Before we can start WildFly, we edit a few things. First, we make sure that we can reach WildFly from other hosts

<interfaces>
    <interface name="management">
        <nic name="eno16777736"/>
    </interface>
    <interface name="public">
        <nic name="eno16777736"/>
    </interface>
    <interface name="unsecure">
        <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
    </interface>
</interfaces>

Next, we add a managment user and an application user, the latter is needed when accessing messaging resources,

<subsystem xmlns="urn:jboss:domain:remoting:4.0">
    <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
</subsystem>

<subsystem xmlns="urn:jboss:domain:messaging-activemq:9.0">
    <server name="default">
        ...
        <security-setting name="#">
            <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
        </security-setting>
        ...
    </server>
</subsystem>

To add users, we can use the add-user utility, for example,

[weblogic@javanode1 bin]$ ./add-user.sh

What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a): b

Enter the details of the new user to add.
Using realm 'ApplicationRealm' as discovered from the existing property files.
Username : employee
Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
- The password should be different from the username
- The password should not be one of the following restricted values {root, admin, administrator}
- The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
Password :
WFLYDM0102: Password should have at least 1 non-alphanumeric symbol.
Are you sure you want to use the password entered yes/no? yes
Re-enter Password :
What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[ ]: guest
About to add user 'employee' for realm 'ApplicationRealm'
Is this correct yes/no? yes
Added user 'employee' to file '/home/weblogic/wildfly/wildfly-19.1.0.Final/standalone/configuration/application-users.properties'
Added user 'employee' to file '/home/weblogic/wildfly/wildfly-19.1.0.Final/domain/configuration/application-users.properties'
Added user 'employee' with groups guest to file '/home/weblogic/wildfly/wildfly-19.1.0.Final/standalone/configuration/application-roles.properties'
Added user 'employee' with groups guest to file '/home/weblogic/wildfly/wildfly-19.1.0.Final/domain/configuration/application-roles.properties'
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? yes
To represent the user add the following to the server-identities definition <secret value="..." />

Note that before we can use the add-user utility we have to provide a JAVA_HOME environment variable. To this end, we edit the common.sh file (located in the $WILDFLY_HOME/bin directory):

#!/bin/sh -x

setModularJdk() { ... }

setDefaultModularJvmOptions() { ... }

# JDK 8
#JAVA_HOME=/u01/app/oracle/weblogic12.2.1/jdk
# JDK 11
JAVA_HOME=/u01/app/oracle/weblogic14.1.1/jdk
export JAVA_HOME

With the management and application user in place, we start the server:

[weblogic@javanode1 bin]$ ./standalone.sh --server-config=standalone-full.xml
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /home/weblogic/wildfly/wildfly-19.1.0.Final

  JAVA: /u01/app/oracle/weblogic14.1.1/jdk/bin/java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED

=========================================================================
...

Next, we add a JMS queue:

[weblogic@javanode1 bin]$ ./jboss-cli.sh --connect --controller=remote+http://javanode1.ms.com:9990
[standalone@javanode1.ms.com:9990 /]

[standalone@javanode1.ms.com:9990 /] jms-queue --help --commands

[standalone@javanode1.ms.com:9990 /] jms-queue add --queue-address=testQueue --entries=jms/queue/test,java:jboss/exported/jms/queue/test

[standalone@javanode1.ms.com:9990 /] /subsystem=messaging-activemq/server=default/jms-queue=testQueue:read-resource
{
    "outcome" => "success",
    "result" => {
        "durable" => true,
        "entries" => [
            "jms/queue/test",
            "java:jboss/exported/jms/queue/test"
        ],
        "legacy-entries" => undefined,
        "selector" => undefined
    }
}

<subsystem xmlns="urn:jboss:domain:messaging-activemq:9.0">
    <server name="default">
        ...
        <jms-queue name="testQueue" entries="jms/queue/test java:jboss/exported/jms/queue/test"/>
        ...
        <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
        ...
    </server>
</subsystem>

Note that when accessing resources remotely, the JNDI name has to start with java:jboss/exported (see here for more information).

To test the set-up, we create a JMS client, for example,

package test;

import javax.jms.*;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.util.Properties;

public class Test {
    public static void main(String[] args) {
        final Properties env = new Properties();
        env.put(Context.INITIAL_CONTEXT_FACTORY, "org.wildfly.naming.client.WildFlyInitialContextFactory");
        env.put(Context.PROVIDER_URL, "http-remoting://javanode1.ms.com:8080");
        env.put(Context.SECURITY_PRINCIPAL, "employee");
        env.put(Context.SECURITY_CREDENTIALS, "...");
        env.put("jboss.naming.client.ejb.context", true);

        ConnectionFactory connectionFactory = null;
        Destination destination = null;

        try {
            InitialContext context = new InitialContext(env);

            connectionFactory = (ConnectionFactory) context.lookup("jms/RemoteConnectionFactory");
            destination = (Destination) context.lookup("jms/queue/test");

            System.out.println(connectionFactory);
            System.out.println(destination);

            sendMessage(connectionFactory, destination);
        } catch (NamingException e) {
            e.printStackTrace();
        }
    }

    private static void sendMessage(ConnectionFactory connectionFactory, Destination destination) {
        Connection connection = null;
        Session session = null;
        MessageProducer messageProducer = null;

        try {
            connection = connectionFactory.createConnection("employee", "...");
            session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
            messageProducer = session.createProducer(destination);

            TextMessage text = session.createTextMessage();
            text.setText("Send some useful message");
            messageProducer.send(text);
        } catch (JMSException e) {
            e.printStackTrace();
        } finally {
            if (connection != null) {
                try {
                    connection.close();
                } catch (JMSException f) {
                    f.printStackTrace();
                }
            }
        }

    }
}

When we run the code above, the following is observed

May 21, 2020 9:49:36 AM org.wildfly.naming.client.Version <clinit>
INFO: WildFly Naming version 1.0.10.Final
May 21, 2020 9:49:36 AM org.wildfly.security.Version <clinit>
INFO: ELY00001: WildFly Elytron version 1.9.1.Final
May 21, 2020 9:49:36 AM org.xnio.Xnio <clinit>
INFO: XNIO version 3.7.2.Final
May 21, 2020 9:49:36 AM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.7.2.Final
May 21, 2020 9:49:36 AM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.3.Final
May 21, 2020 9:49:36 AM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 5.0.12.Final
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ActiveMQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=192-168-101-110&blockOnNonDurableSend=false&retryIntervalMultiplier=1-0&maxRetryInterval=2000&producerMaxRate=-1&blockOnDurableSend=true&connectionTTL=60000&compressLargeMessage=false&reconnectAttempts=0&ignoreJTA=false&cacheLargeMessagesClient=false&scheduledThreadPoolMaxSize=5&useGlobalPools=true&callFailoverTimeout=60000&initialConnectAttempts=1&clientFailureCheckPeriod=30000&blockOnAcknowledge=false&consumerWindowSize=1048576&minLargeMessageSize=102400&useTopologyForLoadBalancing=true&finalizeChecks=false&autoGroup=false&threadPoolMaxSize=30&confirmationWindowSize=-1&transactionBatchSize=1048576&callTimeout=30000&preAcknowledge=false&enable1xPrefixes=true&cacheDestinations=false&connectionLoadBalancingPolicyClassName=org-apache-activemq-artemis-api-core-client-loadbalance-RoundRobinConnectionLoadBalancingPolicy&dupsOKBatchSize=1048576&incomingInterceptorList=&initialMessagePacketSize=1500&consumerMaxRate=-1&enableSharedClientID=true&HA=false&retryInterval=2000&factoryType=0&failoverOnInitialConnection=false&outgoingInterceptorList=&producerWindowSize=65536], discoveryGroupConfiguration=null], clientID=null, consumerWindowSize = 1048576, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=falseEnableSharedClientID=true]
ActiveMQQueue[jms.queue.testQueue]

Process finished with exit code 0

Set-Up WebLogic

Information to install and configure WebLogic can be found here. In our case, we are going to use Ansible to install and configure WebLogic, an example of which can found here. When using JDK11 the JRE directoy no longer exists and the java.security file is now located in the $JAVA_HOME/conf/security directory.

# /etc/ansible/roles/wls_application_vars/defaults/main.yml


##### SetEnvironmentVariables.sh #####
root_directory:            "/u01"

# Directory where the software to be installed is located
software_directory:        "/mnt/hgfs/vmware_temp"
    
# Name of JVM file that is used in the installation
#jvm_file_name:             "jdk-8u231-linux-x64.tar.gz"
jvm_file_name:             "jdk-11.0.6_linux-x64_bin.tar.gz"
    
# Name of the WebLogic file that is used in the installation
weblogic_file_name:        "fmw_14.1.1.0.0_wls_generic.jar"
...
##### files #####
set_user_overrides_sh:      "{{ domain_configuration_home }}/bin/setUserOverrides.sh"
set_domain_env_sh:          "{{ domain_configuration_home }}/bin/setDomainEnv.sh"
config_xml:                 "{{ domain_configuration_home }}/config/config.xml"
admin_server_lock_file:     "{{ domain_configuration_home }}/servers/{{ admin_server_name }}/data/nodemanager/{{ admin_server_name }}.lck"
#java_security:              "{{ java_home }}/jre/lib/security/java.security"
java_security:              "{{ java_home }}/conf/security/java.security"
...
##### domain parameters #####
# showAvailableTemplates('false','true','false')
weblogic_template_name:                          "Basic WebLogic Server Domain"
#weblogic_template_version:                       "12.2.1.3"
weblogic_template_version:                       "14.1.1.0.0"
...

When the Ansible playbooks are run the following output is observed

[ansible@datanode ansible]$ ansible-playbook wls_admin_server.yml --inventory=inventories/production/wls_hosts.cfg

PLAY [wls_admin_server] ***********************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************
ok: [javanode1]

TASK [wls_install : create root directory] ****************************************************************************************************************************************
ok: [javanode1]

TASK [wls_install : create directories] *******************************************************************************************************************************************
changed: [javanode1] => (item=/u01/app/oracle/weblogic14.1.1/scripts)
changed: [javanode1] => (item=/mnt/hgfs/vmware_temp/application_files)
changed: [javanode1] => (item=/mnt/hgfs/vmware_temp/application_files/templates)
skipping: [javanode1] => (item=)
changed: [javanode1] => (item=/mnt/hgfs/vmware_temp/file_stores)
skipping: [javanode1] => (item=)

TASK [wls_install : copy shell scripts] *******************************************************************************************************************************************
changed: [javanode1]

TASK [wls_install : copy python scripts] ******************************************************************************************************************************************
changed: [javanode1]

TASK [wls_install : create SetEnvironmentVariables.sh] ****************************************************************************************************************************
changed: [javanode1]

TASK [wls_install : create environment.properties] ********************************************************************************************************************************
changed: [javanode1]

TASK [wls_install : create silent install file] ***********************************************************************************************************************************
ok: [javanode1]

TASK [wls_install : create oraInst.loc] *******************************************************************************************************************************************
ok: [javanode1]

TASK [wls_install : install java virtual machine] *********************************************************************************************************************************
changed: [javanode1]

TASK [wls_install : rename java virtual machine directory] ************************************************************************************************************************
changed: [javanode1]

TASK [wls_install : set entropy gathering device] *********************************************************************************************************************************
changed: [javanode1]

TASK [wls_install : install weblogic server] **************************************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create create_basic_domain.py] ********************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create basic domain] ******************************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : debug basic domain creation] **********************************************************************************************************************
ok: [javanode1] => {
    "created_basic_domain.stdout_lines": [
        "",
        "Initializing WebLogic Scripting Tool (WLST) ...",
        "",
        "Welcome to WebLogic Server Administration Scripting Shell",
        "",
        "Type help() for help on available commands",
        "",
        "CREATING VARIABLES FROM PROPERTIES FILE",
        "CREATING VARIABLES FROM ENVIRONMENT VARIABLES",
        "CREATING ADDITIONAL VARIABLES",
        "CREATING DOMAIN",
        "SELECTING TEMPLATES",
        "LOADING TEMPLATES",
        "SETTING DOMAIN OPTIONS",
        "CHANGING ADMIN USER",
        "SAVING DOMAIN",
        "READING DOMAIN: /u01/app/oracle/weblogic14.1.1/configuration/domains/tryout_domain",
        "SET NODE MANAGER CREDENTIALS",
        "DISABLE HOSTNAME VERIFICATION",
        "SET UP LDAP CONFIGURATION",
        "ADJUSTING DOMAIN LOG",
        "ADJUSTING ADMIN SERVER LOG",
        "SAVING CHANGES",
        "CREATING ADDITIONAL CONFIGURATION FILES",
        "- CREATING boot.properties",
        "WRITING FILE boot.properties"
    ]
}

TASK [wls_config_admin_server : create application home directory] ****************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create nodemanager.properties] ********************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : disable derby flag] *******************************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create setUserOverrides.sh] ***********************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create nodemanager init script] *******************************************************************************************************************
skipping: [javanode1]

TASK [wls_config_admin_server : create nodemanager systemd script] ****************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : enable and start nodemanager service] *************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create store_user_config.py for node manager] *****************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create user configuration file and associated key file for node manager] **************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : debug user configuration file and associated key file creation for node manager] ******************************************************************
ok: [javanode1] => {
    "created_node_manager_user_configuration_file.stdout_lines": [
        "",
        "Initializing WebLogic Scripting Tool (WLST) ...",
        "",
        "Welcome to WebLogic Server Administration Scripting Shell",
        "",
        "Type help() for help on available commands",
        "",
        "CREATING VARIABLES FROM PROPERTIES FILE",
        "CREATING VARIABLES FROM ENVIRONMENT VARIABLES",
        "CREATING ADDITIONAL VARIABLES",
        "CREATING USER CONFIG FILES",
        "- connecting to node manager",
        "Connecting to Node Manager ...",
        "Successfully Connected to Node Manager.",
        "- creating user config file and associated key file for node manager",
        "Currently connected to Node Manager to monitor the domain tryout_domain.",
        "Creating the key file can reduce the security of your system if it is not kept in a secured location after it is created. Creating new key...",
        "The username and password that were used for this WebLogic NodeManager connection are stored in /u01/app/oracle/weblogic14.1.1/scripts/.node_manager_config_file.properties and /u01/app/oracle/weblogic14.1.1/scripts/.node_manager_key_file.properties.",
        "- disconnecting from node manager",
        "Successfully disconnected from Node Manager."
    ]
}

TASK [wls_config_admin_server : start admin server] *******************************************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create store_user_config.py for admin server] *****************************************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : create user configuration file and associated key file for admin server] **************************************************************************
changed: [javanode1]

TASK [wls_config_admin_server : debug user configuration file and associated key file creation for admin server] ******************************************************************
ok: [javanode1] => {
    "created_admin_server_user_configuration_file.stdout_lines": [
        "",
        "Initializing WebLogic Scripting Tool (WLST) ...",
        "",
        "Welcome to WebLogic Server Administration Scripting Shell",
        "",
        "Type help() for help on available commands",
        "",
        "CREATING VARIABLES FROM PROPERTIES FILE",
        "CREATING VARIABLES FROM ENVIRONMENT VARIABLES",
        "CREATING ADDITIONAL VARIABLES",
        "CREATING USER CONFIG FILES",
        "- connecting to admin server",
        "Connecting to t3://javanode1.ms.com:7001 with userid weblogic ...",
        "Successfully connected to Admin Server \"AdminServer\" that belongs to domain \"tryout_domain\".",
        "",
        "Warning: An insecure protocol was used to connect to the server. ",
        "To ensure on-the-wire security, the SSL port or Admin port should be used instead.",
        "",
        "- creating user config file and associated key file for admin server",
        "Creating the key file can reduce the security of your system if it is not kept in a secured location after it is created. Creating new key...",
        "The username and password that were used for this WebLogic Server connection are stored in /u01/app/oracle/weblogic14.1.1/scripts/.admin_server_config_file.properties and /u01/app/oracle/weblogic14.1.1/scripts/.admin_server_key_file.properties.",
        "- disconnecting from admin server",
        "Disconnected from weblogic server: AdminServer"
    ]
}

TASK [wls_config_admin_server : delete temporary files that contain usernames and passwords] **************************************************************************************
changed: [javanode1] => (item=/mnt/hgfs/vmware_temp/application_files/create_basic_domain.py)
changed: [javanode1] => (item=/mnt/hgfs/vmware_temp/application_files/store_user_config.py)

TASK [wls_config_deployment_environment : create create_deployment_environment.py] ************************************************************************************************
changed: [javanode1]

TASK [wls_config_deployment_environment : create deployment environment] **********************************************************************************************************
changed: [javanode1]

TASK [wls_config_deployment_environment : debug deployment environment creation] **************************************************************************************************
ok: [javanode1] => {
    "created_deployment_environment.stdout_lines": [
        "",
        "Initializing WebLogic Scripting Tool (WLST) ...",
        "",
        "Welcome to WebLogic Server Administration Scripting Shell",
        "",
        "Type help() for help on available commands",
        "",
        "CREATING VARIABLES FROM PROPERTIES FILE",
        "CREATING VARIABLES FROM ENVIRONMENT VARIABLES",
        "CREATING ADDITIONAL VARIABLES",
        "CONFIGURING DOMAIN",
        "CONNECTING TO ADMIN SERVER",
        "- using config and key file",
        "Connecting to t3://javanode1.ms.com:7001 with userid weblogic ...",
        "Successfully connected to Admin Server \"AdminServer\" that belongs to domain \"tryout_domain\".",
        "",
        "Warning: An insecure protocol was used to connect to the server. ",
        "To ensure on-the-wire security, the SSL port or Admin port should be used instead.",
        "",
        "START EDIT MODE",
        "CREATING DYNAMIC CLUSTER",
        "- creating dynamic cluster application_cluster",
        "- creating server template application_cluster_server_template",
        " - configuring overload protection",
        " - configuring logging",
        "- adding server template to the cluster application_cluster",
        "CREATING MACHINES",
        "- creating unix machine machine_javanode1",
        " - configuring node manager",
        "- creating unix machine machine_javanode2",
        " - configuring node manager",
        "SAVE AND ACTIVATE CHANGES",
        "- changes that are activated:",
        " - bean: com.bea:Name=tryout_domain,Type=Domain, attribute: Clusters, modification: create, old value: None, new value: application_cluster",
        " - bean: com.bea:Name=application_cluster,Type=Cluster, attribute: NumberOfServersInClusterAddress, modification: modify, old value: None, new value: 2",
        " - bean: com.bea:Name=application_cluster,Type=Cluster, attribute: WeblogicPluginEnabled, modification: modify, old value: None, new value: True",
        " - bean: com.bea:Name=tryout_domain,Type=Domain, attribute: Machines, modification: create, old value: None, new value: machine_javanode1",
        " - bean: com.bea:Name=machine_javanode1,Type=UnixMachine, attribute: PostBindUID, modification: modify, old value: None, new value: weblogic",
        " - bean: com.bea:Name=machine_javanode1,Type=UnixMachine, attribute: PostBindUIDEnabled, modification: modify, old value: None, new value: True",
        " - bean: com.bea:Name=machine_javanode1,Type=UnixMachine, attribute: PostBindGIDEnabled, modification: modify, old value: None, new value: True",
        " - bean: com.bea:Name=machine_javanode1,Type=UnixMachine, attribute: PostBindGID, modification: modify, old value: None, new value: javainstall",
        " - bean: com.bea:Name=tryout_domain,Type=Domain, attribute: Machines, modification: create, old value: None, new value: machine_javanode2",
        " - bean: com.bea:Name=machine_javanode2,Type=UnixMachine, attribute: PostBindUID, modification: modify, old value: None, new value: weblogic",
        " - bean: com.bea:Name=machine_javanode2,Type=UnixMachine, attribute: PostBindUIDEnabled, modification: modify, old value: None, new value: True",
        " - bean: com.bea:Name=machine_javanode2,Type=UnixMachine, attribute: PostBindGIDEnabled, modification: modify, old value: None, new value: True",
        " - bean: com.bea:Name=machine_javanode2,Type=UnixMachine, attribute: PostBindGID, modification: modify, old value: None, new value: javainstall",
        " - bean: com.bea:Name=tryout_domain,Type=Domain, attribute: ServerTemplates, modification: create, old value: None, new value: application_cluster_server_template",
        " - bean: com.bea:Name=application_cluster_server_template,Type=ServerTemplate, attribute: Cluster, modification: modify, old value: None, new value: application_cluster",
        " - bean: com.bea:Name=application_cluster_server_template,Type=ServerTemplate, attribute: ListenPort, modification: modify, old value: None, new value: 9001",
        "- status of this activation per server:",
        " - server: AdminServer, status: committed",
        "- checking if admin server needs to be restarted, and checking if managed servers are affected by the change, if so these will be shutdown",
        "START EDIT MODE",
        "CREATING CLUSTERED MESSAGING MODULE",
        "- creating file store application_cluster_filestore",
        "- creating jms server application_cluster_jms_server",
        "- creating jms system module",
        " - creating sub deployment jms_system_resource_module_sub_deployment",
        " - setting sub deployment targets based on available jms server",
        "CREATING MESSAGING RESOURCES",
        "- creating connection factories",
        " - creating connection factory connection_factory",
        "- creating uniform distributed queues",
        " - creating uniform distributed queue company_distributed_queue",
        " - creating error destination company_distributed_error_queue",
        "CREATING DATA SOURCES",
        "- creating jdbc system resource",
        " - setting data source parameters",
        " - setting jdbc driver parameters",
        " - setting jdbc connection pool parameters",
        "SAVE AND ACTIVATE CHANGES",
        "- changes that are activated:",
        " - bean: com.bea:Name=tryout_domain,Type=Domain, attribute: FileStores, modification: create, old value: None, new value: application_cluster_filestore",
        " - bean: com.bea:Name=application_cluster_filestore,Type=FileStore, attribute: Directory, modification: modify, old value: None, new value: /mnt/hgfs/vmware_temp/file_stores",
        " - bean: com.bea:Name=application_cluster_filestore,Type=FileStore, attribute: Targets, modification: add, old value: None, new value: application_cluster",
        " - bean: com.bea:Name=tryout_domain,Type=Domain, attribute: JDBCSystemResources, modification: create, old value: None, new value: data_source",
        " - bean: com.bea:Name=data_source,Type=JDBCSystemResource, attribute: Targets, modification: add, old value: None, new value: application_cluster",
        " - bean: com.bea:Name=tryout_domain,Type=Domain, attribute: JMSServers, modification: create, old value: None, new value: application_cluster_jms_server",
        " - bean: com.bea:Name=application_cluster_jms_server,Type=JMSServer, attribute: PersistentStore, modification: modify, old value: None, new value: application_cluster_filestore",
        " - bean: com.bea:Name=application_cluster_jms_server,Type=JMSServer, attribute: Targets, modification: add, old value: None, new value: application_cluster",
        " - bean: com.bea:Name=tryout_domain,Type=Domain, attribute: JMSSystemResources, modification: create, old value: None, new value: jms_system_resource_module",
        " - bean: com.bea:Name=jms_system_resource_module,Type=JMSSystemResource, attribute: Targets, modification: add, old value: None, new value: application_cluster",
        "- status of this activation per server:",
        " - server: AdminServer, status: committed",
        "- checking if admin server needs to be restarted, and checking if managed servers are affected by the change, if so these will be shutdown",
        "DISCONNECT FROM THE ADMIN SERVER",
        "Disconnected from weblogic server: AdminServer"
    ]
}

TASK [wls_config_deployment_environment : remove old template] ********************************************************************************************************************
changed: [javanode1]

TASK [wls_config_deployment_environment : create new template] ********************************************************************************************************************
changed: [javanode1]

TASK [wls_config_deployment_environment : delete temporary files that contain usernames and passwords] ****************************************************************************
changed: [javanode1] => (item=/mnt/hgfs/vmware_temp/application_files/create_deployment_environment.py)

TASK [wls_application_config : create application_config.py] **********************************************************************************************************************
changed: [javanode1]

TASK [wls_application_config : execute application_config.py] *********************************************************************************************************************
changed: [javanode1]

TASK [wls_application_config : debug application config] **************************************************************************************************************************
ok: [javanode1] => {
    "executed_application_config.stdout_lines": [
        "",
        "Initializing WebLogic Scripting Tool (WLST) ...",
        "",
        "Welcome to WebLogic Server Administration Scripting Shell",
        "",
        "Type help() for help on available commands",
        "",
        "CREATING VARIABLES FROM PROPERTIES FILE",
        "CREATING VARIABLES FROM ENVIRONMENT VARIABLES",
        "CREATING ADDITIONAL VARIABLES",
        "CONFIGURING DOMAIN",
        "CONNECTING TO ADMIN SERVER",
        "- using config and key file",
        "Connecting to t3://javanode1.ms.com:7001 with userid weblogic ...",
        "Successfully connected to Admin Server \"AdminServer\" that belongs to domain \"tryout_domain\".",
        "",
        "Warning: An insecure protocol was used to connect to the server. ",
        "To ensure on-the-wire security, the SSL port or Admin port should be used instead.",
        "",
        "START EDIT MODE",
        "CREATING WORK MANAGERS",
        "CONFIGURING ENVIRONMENT",
        "SAVE AND ACTIVATE CHANGES",
        "- no changes",
        "CREATING USERS AND GROUPS",
        "DISCONNECT FROM THE ADMIN SERVER",
        "Disconnected from weblogic server: AdminServer"
    ]
}

TASK [wls_application_config : delete temporary files that contain usernames and passwords] ***************************************************************************************
changed: [javanode1] => (item=/mnt/hgfs/vmware_temp/application_files/application_config.py)

PLAY RECAP ************************************************************************************************************************************************************************
javanode1                  : ok=40   changed=31   unreachable=0    failed=0

[ansible@datanode ansible]$ ansible-playbook wls_managed_servers.yml --inventory=inventories/production/wls_hosts.cfg

PLAY [wls_managed_servers] ********************************************************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************************************************
ok: [javanode2]

TASK [wls_install : create root directory] ****************************************************************************************************************************************
ok: [javanode2]

TASK [wls_install : create directories] *******************************************************************************************************************************************
changed: [javanode2] => (item=/u01/app/oracle/weblogic14.1.1/scripts)
changed: [javanode2] => (item=/mnt/hgfs/vmware_temp/application_files)
changed: [javanode2] => (item=/mnt/hgfs/vmware_temp/application_files/templates)
skipping: [javanode2] => (item=)
changed: [javanode2] => (item=/mnt/hgfs/vmware_temp/file_stores)
skipping: [javanode2] => (item=)

TASK [wls_install : copy shell scripts] *******************************************************************************************************************************************
changed: [javanode2]

TASK [wls_install : copy python scripts] ******************************************************************************************************************************************
changed: [javanode2]

TASK [wls_install : create SetEnvironmentVariables.sh] ****************************************************************************************************************************
changed: [javanode2]

TASK [wls_install : create environment.properties] ********************************************************************************************************************************
changed: [javanode2]

TASK [wls_install : create silent install file] ***********************************************************************************************************************************
ok: [javanode2]

TASK [wls_install : create oraInst.loc] *******************************************************************************************************************************************
ok: [javanode2]

TASK [wls_install : install java virtual machine] *********************************************************************************************************************************
changed: [javanode2]

TASK [wls_install : rename java virtual machine directory] ************************************************************************************************************************
changed: [javanode2]

TASK [wls_install : set entropy gathering device] *********************************************************************************************************************************
changed: [javanode2]

TASK [wls_install : install weblogic server] **************************************************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : create domain] *********************************************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : create application home directory] *************************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : create nodemanager.properties] *****************************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : disable derby flag] ****************************************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : create setUserOverrides.sh] ********************************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : create nodemanager init script] ****************************************************************************************************************
skipping: [javanode2]

TASK [wls_config_managed_servers : create nodemanager systemd script] *************************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : enable and start nodemanager service] **********************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : create store_user_config.py for node manager] **************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : create user configuration file and associated key file for node manager] ***********************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : debug user configuration file and associated key file creation for node manager] ***************************************************************
ok: [javanode2] => {
    "created_node_manager_user_configuration_file.stdout_lines": [
        "",
        "Initializing WebLogic Scripting Tool (WLST) ...",
        "",
        "Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.",
        "",
        "Welcome to WebLogic Server Administration Scripting Shell",
        "",
        "Type help() for help on available commands",
        "",
        "CREATING VARIABLES FROM PROPERTIES FILE",
        "CREATING VARIABLES FROM ENVIRONMENT VARIABLES",
        "CREATING ADDITIONAL VARIABLES",
        "CREATING USER CONFIG FILES",
        "- connecting to node manager",
        "Connecting to Node Manager ...",
        "Successfully Connected to Node Manager.",
        "- creating user config file and associated key file for node manager",
        "Currently connected to Node Manager to monitor the domain tryout_domain.",
        "Creating the key file can reduce the security of your system if it is not kept in a secured location after it is created. Creating new key...",
        "The username and password that were used for this WebLogic NodeManager connection are stored in /u01/app/oracle/weblogic14.1.1/scripts/.node_manager_config_file.properties and /u01/app/oracle/weblogic14.1.1/scripts/.node_manager_key_file.properties.",
        "- disconnecting from node manager",
        "Successfully disconnected from Node Manager."
    ]
}

TASK [wls_config_managed_servers : create store_user_config.py for admin server] **************************************************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : create user configuration file and associated key file for admin server] ***********************************************************************
changed: [javanode2]

TASK [wls_config_managed_servers : debug user configuration file and associated key file creation for admin server] ***************************************************************
ok: [javanode2] => {
    "created_admin_server_user_configuration_file.stdout_lines": [
        "",
        "Initializing WebLogic Scripting Tool (WLST) ...",
        "",
        "Welcome to WebLogic Server Administration Scripting Shell",
        "",
        "Type help() for help on available commands",
        "",
        "CREATING VARIABLES FROM PROPERTIES FILE",
        "CREATING VARIABLES FROM ENVIRONMENT VARIABLES",
        "CREATING ADDITIONAL VARIABLES",
        "CREATING USER CONFIG FILES",
        "- connecting to admin server",
        "Connecting to t3://javanode1.ms.com:7001 with userid weblogic ...",
        "Successfully connected to Admin Server \"AdminServer\" that belongs to domain \"tryout_domain\".",
        "",
        "Warning: An insecure protocol was used to connect to the server. ",
        "To ensure on-the-wire security, the SSL port or Admin port should be used instead.",
        "",
        "- creating user config file and associated key file for admin server",
        "Creating the key file can reduce the security of your system if it is not kept in a secured location after it is created. Creating new key...",
        "The username and password that were used for this WebLogic Server connection are stored in /u01/app/oracle/weblogic14.1.1/scripts/.admin_server_config_file.properties and /u01/app/oracle/weblogic14.1.1/scripts/.admin_server_key_file.properties.",
        "- disconnecting from admin server",
        "Disconnected from weblogic server: AdminServer"
    ]
}

TASK [wls_config_managed_servers : delete temporary files that contain usernames and passwords] ***********************************************************************************
changed: [javanode2] => (item=/mnt/hgfs/vmware_temp/application_files/store_user_config.py)

TASK [wls_application_config : create application_config.py] **********************************************************************************************************************
changed: [javanode2]

TASK [wls_application_config : execute application_config.py] *********************************************************************************************************************
changed: [javanode2]

TASK [wls_application_config : debug application config] **************************************************************************************************************************
ok: [javanode2] => {
    "executed_application_config.stdout_lines": [
        "",
        "Initializing WebLogic Scripting Tool (WLST) ...",
        "",
        "Welcome to WebLogic Server Administration Scripting Shell",
        "",
        "Type help() for help on available commands",
        "",
        "CREATING VARIABLES FROM PROPERTIES FILE",
        "CREATING VARIABLES FROM ENVIRONMENT VARIABLES",
        "CREATING ADDITIONAL VARIABLES",
        "CONFIGURING DOMAIN",
        "CONNECTING TO ADMIN SERVER",
        "- using config and key file",
        "Connecting to t3://javanode1.ms.com:7001 with userid weblogic ...",
        "Successfully connected to Admin Server \"AdminServer\" that belongs to domain \"tryout_domain\".",
        "",
        "Warning: An insecure protocol was used to connect to the server. ",
        "To ensure on-the-wire security, the SSL port or Admin port should be used instead.",
        "",
        "START EDIT MODE",
        "CREATING WORK MANAGERS",
        "CONFIGURING ENVIRONMENT",
        "SAVE AND ACTIVATE CHANGES",
        "- no changes",
        "CREATING USERS AND GROUPS",
        "- user employee already exists and will not be created",
        "- user manager already exists and will not be created",
        "DISCONNECT FROM THE ADMIN SERVER",
        "Disconnected from weblogic server: AdminServer"
    ]
}

TASK [wls_application_config : delete temporary files that contain usernames and passwords] ***************************************************************************************
changed: [javanode2] => (item=/mnt/hgfs/vmware_temp/application_files/application_config.py)

PLAY RECAP ************************************************************************************************************************************************************************
javanode2                  : ok=31   changed=24   unreachable=0    failed=0

Foreign Server

WebLogic JMS allows us to reference third-party JMS providers within a local WebLogic Server JNDI tree. With Foreign Server resources in JMS modules, we can quickly map a foreign JMS provider so that its associated connection factories and destinations appear in the WebLogic JNDI tree as local JMS objects. More information can be found here.

To configure a foreign server we follow the steps presented here. A few things are worth mentioning:

If the Foreign JMS Server expects a user credential to be passed when making a JMS connection then it should be defined under the foreign connection factory configuration.
Similarly if the Foreign JMS Server expects a user credential to be passed even for JNDI lookup then the user credentials should be configured under JNDI as well. Note that the username is to be specified using a JNDI property, in this case the JNDI property java.naming.security.principal. Also provide any other properties that your JMS provider expects under JNDI Properties. As we are not accessing EJBs, we do not need to specify the jboss.naming.client.ejb.context property (see here for more information).
To be able to use the WildFlyInitialContextFactory and the http-remoting protocol, we need to add the $WILDFLY_HOME/bin/client/jboss-client.jar to WebLogic's class path. This can be accomplished by adding the jar file to WebLogic's $DOMAIN_HOME/lib directory.

The configuration of the foreign server looks as follows:

# machine
<machine xsi:type="unix-machineType">
    <name>machine_javanode1</name>
    <node-manager>
        <nm-type>Plain</nm-type>
        <listen-address>javanode1.ms.com</listen-address>
        <listen-port>5550</listen-port>
    </node-manager>
    <post-bind-uid-enabled>true</post-bind-uid-enabled>
    <post-bind-uid>weblogic</post-bind-uid>
    <post-bind-gid-enabled>true</post-bind-gid-enabled>
    <post-bind-gid>javainstall</post-bind-gid>
</machine>

# managed server
<server>
    <name>testserver</name>
    <ssl>
        <enabled>false</enabled>
    </ssl>
    <machine>machine_javanode1</machine>
    <listen-port>10080</listen-port>
    <listen-port-enabled>true</listen-port-enabled>
    <cluster xsi:nil="true"></cluster>
    <web-server>
        <web-server-log>
            <number-of-files-limited>false</number-of-files-limited>
        </web-server-log>
    </web-server>
    <listen-address>javanode1.ms.com</listen-address>
    <java-compiler>javac</java-compiler>
    <client-cert-proxy-enabled>false</client-cert-proxy-enabled>
    <server-diagnostic-config>
        <wldf-diagnostic-volume>Low</wldf-diagnostic-volume>
    </server-diagnostic-config>
</server>

# jms system resource
<jms-system-resource>
    <name>testmodule</name>
    <target>testserver</target>
    <descriptor-file-name>jms/testmodule-jms.xml</descriptor-file-name>
</jms-system-resource>

# foreign server
<weblogic-jms ...>
    <foreign-server name="testforeignserver">
        <default-targeting-enabled>true</default-targeting-enabled>
        <foreign-destination name="TestForeignDestination">
            <local-jndi-name>jms/TestForeignDestination</local-jndi-name>
            <remote-jndi-name>jms/queue/test</remote-jndi-name>
        </foreign-destination>
        <foreign-connection-factory name="TestForeignConnectionFactory-0">
            <local-jndi-name>jms/TestForeignConnectionFactory-0</local-jndi-name>
            <remote-jndi-name>jms/RemoteConnectionFactory</remote-jndi-name>
            <username>employee</username>
            <password-encrypted>...</password-encrypted>
            <connection-health-checking>enabled</connection-health-checking>
        </foreign-connection-factory>
        <initial-context-factory>org.wildfly.naming.client.WildFlyInitialContextFactory</initial-context-factory>
        <connection-url>http-remoting://javanode1.ms.com:8080</connection-url>
        <jndi-properties-credential-encrypted>...</jndi-properties-credential-encrypted>
        <jndi-property>
            <key>java.naming.security.principal</key>
            <value>employee</value>
        </jndi-property>
    </foreign-server>
</weblogic-jms>

Next start the managed server, and check the JNDI tree of the managed server

To consume messages, we create a message-driven bean (example can be found in the JavaEE Tutorial).

package logic;

import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.jms.*;

@MessageDriven(name = "ReceiveMDB", activationConfig = {
        @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
        @ActivationConfigProperty(propertyName = "destinationLookup", propertyValue = "jms/TestForeignDestination"),
        @ActivationConfigProperty(propertyName = "connectionFactoryLookup", propertyValue = "jms/TestForeignConnectionFactory-0")
})
public class ReceiveMDB implements MessageListener {
    @Override
    public void onMessage(Message message) {
        if (message instanceof TextMessage) {
            TextMessage textMessage = (TextMessage) message;
            try {
                System.out.println("RECEIVED TEXT MESSAGE " + textMessage.getText());
            } catch (JMSException e) {
                e.printStackTrace();
            }
        }
    }
}

The ActivationConfigProperty provides the connection information:

destinationType. This property is used to specify whether the message driven bean is intended to be used with a queue or a topic. The value must be either javax.jms.Queue or javax.jms.Topic.
destinationLookup. This property is used to specify the JMS queue or topic from which a JMS message-driven bean is to receive messages.
connectionFactoryLookup. This property is used to specify the JMS connection factory that will be used to connect to the JMS provider from which a JMS message-driven bean is to receive messages.

Start the managed server and deploy the message-driven bean.

Note that by default WebLogic creates 16 consumer, this can edited by using the META-INF/weblogic-ejb-jar.xml, for example,

<weblogic-ejb-jar ...>
    <weblogic-enterprise-bean>
        <ejb-name>ReceiveMDB</ejb-name>
        <message-driven-descriptor>
            <pool>
                <max-beans-in-free-pool>1</max-beans-in-free-pool>
                <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
            </pool>
        </message-driven-descriptor>
    </weblogic-enterprise-bean>
</weblogic-ejb-jar>

Messaging Bridge

The Messaging Bridge is a forwarding mechanism that provides interoperability between WebLogic JMS implementations. A messaging bridge also provides interoperability between JMS and other messaging products.

Use the Messaging Bridge to integrate your messaging applications between:

Any two implementations of WebLogic JMS, including those from separate releases of WebLogic Server.
WebLogic JMS implementations that reside in separate WebLogic domains.
WebLogic JMS and a third-party JMS product (for example, MQSeries).

A messaging bridge instance forwards messages between a pair of bridge source and target destinations. These destinations are mapped to a pair of bridge source and target destinations. The messaging bridge reads messages from the source bridge destination and forwards those messages to the target bridge destination. For WebLogic JMS and third-party JMS products, a messaging bridge communicates with source and target destinations using the Java EE Connector Architecture (JCA) resource adapters provided with WebLogic Server.

Untarget the JMS system resource that contains the foreign server, and stop the application containing the message-driven bean (first we just want to check the message forwarding).

To configure a messaging bridge we follow the steps presented here. The configuration of the messaging bridge looks as follows:

# target JMS resources
# persistent file store
<file-store>
    <name>teststore</name>
    <directory>/mnt/hgfs/vmware_temp/file_stores</directory>
    <target>testserver</target>
</file-store>

# jms server
<jms-server>
    <name>testjmsserver</name>
    <target>testserver</target>
    <persistent-store>teststore</persistent-store>
</jms-server>

# jms system resource
<jms-system-resource>
    <name>testbridgemodule</name>
    <target>testserver</target>
    <sub-deployment>
      <name>testsubdeployment</name>
      <target>testjmsserver</target>
    </sub-deployment>
    <descriptor-file-name>jms/testbridgemodule-jms.xml</descriptor-file-name>
</jms-system-resource>

# jms resources ( queue and connection factory )
<weblogic-jms ...>
  <connection-factory name="ConnectionFactory">
    <default-targeting-enabled>true</default-targeting-enabled>
    <jndi-name>jms/TestForeignConnectionFactory-0</jndi-name>
    <client-params>
      <client-id-policy>Restricted</client-id-policy>
      <subscription-sharing-policy>Exclusive</subscription-sharing-policy>
      <messages-maximum>10</messages-maximum>
    </client-params>
    <transaction-params>
      <xa-connection-factory-enabled>true</xa-connection-factory-enabled>
    </transaction-params>
    <security-params>
      <attach-jmsx-user-id>false</attach-jmsx-user-id>
    </security-params>
  </connection-factory>
  <queue name="Queue">
    <sub-deployment-name>testsubdeployment</sub-deployment-name>
    <jndi-name>jms/TestForeignDestination</jndi-name>
  </queue>
</weblogic-jms>

# bridge
# source destination (WildFly)
<jms-bridge-destination>
    <name>SourceDestination</name>
    <adapter-jndi-name>eis.jms.WLSConnectionFactoryJNDINoTX</adapter-jndi-name>
    <user-name>employee</user-name>
    <user-password-encrypted>{AES256}g3eYe06CVfMSxyGpK0IuavG2j3bR2c2+LGy/QEsaaHk=</user-password-encrypted>
    <connection-factory-jndi-name>jms/RemoteConnectionFactory</connection-factory-jndi-name>
    <initial-context-factory>org.wildfly.naming.client.WildFlyInitialContextFactory</initial-context-factory>
    <connection-url>http-remoting://javanode1.ms.com:8080</connection-url>
    <destination-jndi-name>jms/queue/test</destination-jndi-name>
    <destination-type>Queue</destination-type>
</jms-bridge-destination>

# target destination (WebLogic)
<jms-bridge-destination>
    <name>TargetDestination</name>
    <adapter-jndi-name>eis.jms.WLSConnectionFactoryJNDINoTX</adapter-jndi-name>
    <connection-factory-jndi-name>jms/TestForeignConnectionFactory-0</connection-factory-jndi-name>
    <connection-url>t3://javanode1.ms.com:10080</connection-url>
    <destination-jndi-name>jms/TestForeignDestination</destination-jndi-name>
</jms-bridge-destination>

# messaging bridge
<messaging-bridge>
    <name>Bridge</name>
    <target>testserver</target>
    <source-destination>SourceDestination</source-destination>
    <target-destination>TargetDestination</target-destination>
    <selector></selector>
    <quality-of-service>Atmost-once</quality-of-service>
    <started>true</started>
</messaging-bridge>

Check the WildFly and WebLogic console to see if the set-up is working

Send a message

Start the application containing the message-driven bean to consume the message, and check the logging (for no particular reason).

# $WILDFLY_HOME/standalone/log/server.log
2020-05-21 09:54:48,155 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
2020-05-21 09:54:48,158 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://192.168.101.110:9990/management
2020-05-21 09:54:48,158 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://192.168.101.110:9990
2020-05-21 09:54:48,158 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 19.1.0.Final (WildFly Core 11.1.1.Final) started in 5487ms - Started 359 of 626 services (393 services are lazy, passive or on-demand)
2020-05-21 09:55:01,243 INFO  [org.wildfly.naming] (default task-1) WildFly Naming version 1.0.12.Final

# $DOMAIN_HOME/servers/testserver/logs/testserver.out
<May 21, 2020, 9:58:32,127 AM Central European Summer Time> <Notice> <WebLogicServer> <BEA-000330> <Started the WebLogic Server Managed Server "testserver" for domain "tryout_domain" running in production mode.> 
<May 21, 2020, 9:58:32,133 AM Central European Summer Time> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.> 
<May 21, 2020, 9:58:32,143 AM Central European Summer Time> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.> 
May 21, 2020 9:58:32 AM org.wildfly.naming.client.Version <clinit>
INFO: WildFly Naming version 1.0.12.Final
May 21, 2020 9:58:32 AM org.wildfly.security.Version <clinit>
INFO: ELY00001: WildFly Elytron version 1.11.4.Final
May 21, 2020 9:58:33 AM org.xnio.Xnio <clinit>
INFO: XNIO version 3.7.7.Final
May 21, 2020 9:58:33 AM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.7.7.Final
May 21, 2020 9:58:33 AM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.3.Final
May 21, 2020 9:58:33 AM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 5.0.17.Final
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by javassist.util.proxy.SecurityActions (file:/u01/app/oracle/weblogic14.1.1/installation/oracle_common/modules/org.javassist.javassist.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of javassist.util.proxy.SecurityActions
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
RECEIVED TEXT MESSAGE Send some useful message
RECEIVED TEXT MESSAGE Send some other useful message

References

[1] WildFly Documentation.
[2] WebLogic Documentation.
[3] Java EE Resources.
[4] JavaEE 8 Specification APIs.

Ansibling the Oracle Database

2019-05-20T13:47:00.000+02:00

In this post, we give an example of how to set-up an Oracle Database using Ansible.

Create Ansible Roles

Playbooks are Ansible's configuration, deployment, and orchestration language. While it is possible to write a playbook in one very large file, eventually we will want to reuse files and start to organize things. Roles in Ansible build on the idea of include files and combine them to form clean, reusable abstractions - they allow us to focus more on the big picture and only dive down into the details when needed. Ansible offers a number of modules which are described here. If the need arises, we can also develop our own modules.

In order to install and configure an Oracle Database, we create the following roles

oracle_database_vars - defines the defaults for the used variables.
oracle_database_host - configures the Linux host, i.e., configures SELinux, configures the firewall, installs (necessary) packages, creates a group and a user, modifies kernel parameters (/etc/sysctl.conf), and sets limits on the system resources that can be obtained in a user-session (/etc/security/limits.conf).
oracle_database_install - installs the Oracle Database software.
oracle_database_config - creates a listener and database.

The directory structure for the roles looks as follows

/etc/ansible
    /inventories
        /test
            /group_vars
                database_hosts.yml
            hosts.cfg
    /roles
        /oracle_database_config
            /files
                /oracle/lifecycle
                    start_database.sh
                    stop_database.sh
            /tasks
                main.yml
            /templates
                database_service.j2
                dbca_rsp_12.2.0.j2
                dbca_rsp_19.0.0.j2
                netca_rsp_12.2.0.j2
                netca_rsp_19.0.0.j2
        /oracle_database_host
            /handlers
                main.yml
            /tasks
                main.yml
            /templates
                limits_conf.j2
                sysctl_conf.j2
        /oracle_database_install
            /tasks
                main.yml
            /templates
                db_install_rsp_12.2.0.j2
                db_install_rsp_19.0.0.j2
        /oracle_database_vars
            /defaults
                main.yml
    oracle_database.yml
/mnt/hgfs/vmware_temp
    /database_12.2.0
        /install
        /response
        /rpm
        /sshsetup
        /stage
        runInstaller
        welcome.html
    /database_19.0.0
        V982063-01.zip

Configure (Linux) host

To configure the (Linux) host, we use the following:

Used variables

##### User and group settings #####
# User and Group under which the software needs to be installed
oracle_install_user: "oracle"
oracle_install_password: "set_in_environment_specific_group_vars_file"
oracle_install_group: "oinstall"
oracle_install_group_id: 54321


##### OS settings #####
# number of open files
number_of_open_files: 65536

# number of processes
number_of_processes: 16384

# stack size
stack_size: 10240

# memory that will be used for huge pages
memory_to_reserve_in_mega_bytes: 256

number_of_huge_pages: "{{ memory_to_reserve_in_mega_bytes // 2 }}"
mem_lock: "{{ 2048 * (memory_to_reserve_in_mega_bytes // 2) }}"

On Oracle Linux it is possible to configure the host using the preinstallation RPM.
Install the required packages.
Create the group and user under which the Oracle Database will be installed and configured. Note that the password option of the user module must be encrypted. To create an encrypted password, we can use python -c 'import crypt; print crypt.crypt("user_password", "$id$salt")', in which id can be 1 (MD5), 5 (SHA-256 since glibc 2.7), or 6 (SHA-512 since glibc 2.7), and salt stands for the up to 16 characters following $id$ , more information can be found in the crypt(3) Linux manual page.

Modify kernel parameters

# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536

# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

# semaphores
kernel.sem = 250 32000 100 128
# shared memory
kernel.shmmni = 4096

# file handles
fs.file-max = 6815744

# asynchronous i/o
fs.aio-max-nr = 1048576

# disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

# default range of IP port numbers that are allowed for TCP and UDP traffic
net.ipv4.ip_local_port_range = 9000 65535

# increase TCP max buffer size (depending on the type of NIC and the round-trip time these values can be changed)
# Maximum TCP Receive Window
net.core.rmem_max = 8388608
net.core.rmem_default = 8388608
# Maximum TCP Send Window
net.core.wmem_max = 8388608
net.core.wmem_default = 8388608
#  memory reserved for TCP receive buffers (vector of 3 integers: [min, default, max])
net.ipv4.tcp_rmem = 4096 87380 8388608
# memory reserved for TCP send buffers (vector of 3 integers: [min, default, max])
net.ipv4.tcp_wmem = 4096 87380 8388608

# increase the length of the processor input queue
net.core.netdev_max_backlog = 30000
# maximum amount of memory buffers (could be set equal to net.core.rmem_max and net.core.wmem_max)
net.core.optmem_max = 20480
# socket of the listen backlog
net.core.somaxconn = 1024

# tcp selective acknowledgements (disable them on high-speed networks)
net.ipv4.tcp_sack = 1
net.ipv4.tcp_dsack = 1
# Timestamps add 12 bytes to the TCP header
net.ipv4.tcp_timestamps = 1
# Support for large TCP Windows - Needs to be set to 1 if the Max TCP Window is over 65535
net.ipv4.tcp_window_scaling = 1
# The interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe
net.ipv4.tcp_keepalive_time = 1800
# The interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime
net.ipv4.tcp_keepalive_intvl = 30
# The number of unacknowledged probes to send before considering the connection dead and notifying the application layer
net.ipv4.tcp_keepalive_probes = 5
# The time that must elapse before TCP/IP can release a closed connection and reuse its resources.
net.ipv4.tcp_fin_timeout = 30
# Size of the backlog connections queue.
net.ipv4.tcp_max_syn_backlog = 4096
# The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers.
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

# The percentage of how aggressively memory pages are swapped to disk
vm.swappiness = 0
# The percentage of main memory the pdflush daemon should write data out to the disk.
vm.dirty_background_ratio = 25
# The percentage of main memory the actual disk writes will take place.
vm.dirty_ratio = 20

# set the number of huge pages based on the Hugepagesize, i.e., 2048kB
#vm.nr_hugepages = {{ number_of_huge_pages }}
 
# give permission to the group that runs the process to access the shared memory segment
#vm.hugetlb_shm_group = {{ oracle_install_group_id }}

Modify the limits on the system resources

# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4


# open file descriptors
@{{ oracle_install_group }} soft nofile {{ number_of_open_files }}
@{{ oracle_install_group }} hard nofile {{ number_of_open_files }}

#max number of processes
@{{ oracle_install_group }} soft nproc {{ number_of_processes }}
@{{ oracle_install_group }} hard nproc {{ number_of_processes }}

#max stack size (KB)
@{{ oracle_install_group }} soft stack {{ stack_size }}
@{{ oracle_install_group }} hard stack {{ stack_size }}

# memlock - maximum locked in-memory address space (kB)
#@{{ oracle_install_group }} soft memlock {{ mem_lock }}
#@{{ oracle_install_group }} hard memlock {{ mem_lock }}


# End of file

Configure the Firewall. To this end, we use the firewalld module, and add the listen port, in this case, 1521.

Configure SELinux, to this end, we can use the selinux (and the seboolean, sefcontext, seport) module(s). Note that SELinux is configured by default for the Oracle Database, we can check this by using sesearch --allow | grep -i oracle or a bit more finegrained, for example,

[root@datanode ~]# sesearch --allow --source=httpd_t | grep -i oracle
   allow httpd_t oracle_client_packet_t : packet recv ;
   allow httpd_t oracle_client_packet_t : packet send ;
   allow httpd_t oracle_port_t : tcp_socket name_connect ;

also check which ports are accepted

[root@datanode ~]# semanage port -l | grep oracle_port_t
oracle_port_t                  tcp      1521, 2483, 2484
oracle_port_t                  udp      1521, 2483, 2484

and check file context mappings

[root@datanode ~]# semanage fcontext -l | grep etc_t
/etc/.*                                            all files          system_u:object_r:etc_t:s0
...

[root@datanode ~]# ll /etc/ora*
-rw-r--r--. 1 root   root      56 May  9 13:33 /etc/oraInst.loc
-rw-rw-r--. 1 oracle oinstall 786 May  9 13:34 /etc/oratab

when installing and configuring it is always smart to leave SELinux in permissive mode and check the audit logs (/var/log/audit/audit.log). SELinux troubleshooting is explained in the article: Basic SELinux Troubleshooting in CLI.

To combine the above in an Ansible playbook, we have the following

# /etc/ansible/roles/oracle_database_host/tasks/main.yml

#- name: install oracle database server preinstall
#  yum:
#    name: http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oracle-database-preinstall-19c-1.0-1.el7.x86_64.rpm
#    name: http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oracle-database-server-12cR2-preinstall-1.0-4.el7.x86_64.rpm
#    state: present

- name: install necessary packages
  yum:
    name={{ item }}
    state=present
  loop:
    - binutils
    - compat-libcap1
    - compat-libstdc++-33
    - dos2unix
    - gcc
    - gcc-c++
    - glibc
    - glibc-devel
    - kmod
    - ksh
    - libaio
    - libaio-devel
    - libgcc
    - libstdc++
    - libstdc++-devel
    - libX11
    - libXau
    - libxcb
    - libXext
    - libXi
    - libXrender
    - libXtst
    - lsof
    - ksh
    - make
    - net-tools
    - perl
    - smartmontools
    - strace
    - sysstat
    - unzip

- name: create group
  group:
    name={{ oracle_install_group }}
    gid={{ oracle_install_group_id }}
    state=present
    
- name: create user
  user:
    name={{ oracle_install_user }}
    state=present
    createhome=yes
    group={{ oracle_install_group_id }}
    password={{ oracle_install_password }}

# kernel parameters
- name: set sysctl /etc/sysctl.d/10-default_sysctl.conf
  template: 
    src=roles/oracle_database_host/templates/sysctl_conf.j2 
    dest=/etc/sysctl.d/10-default_sysctl.conf
    owner=root
    group=root
    mode=0644
  register: sysctl_changed
  
- name: run sysctl -p
  command: /bin/sh -c "/sbin/sysctl -p"
  when: sysctl_changed is changed
    
# linux resource limits
- name: set /etc/security/limits.conf
  template: 
    src=roles/oracle_database_host/templates/limits_conf.j2 
    dest=/etc/security/limits.conf
    owner=root
    group=root
    mode=0644
    
# configure firewall
- name: check if iptables exists
  stat: path=/usr/lib/systemd/system/iptables.service
  register: iptables_status
    
- name: disable iptables
  service:
    name=iptables
    state=stopped
    enabled=no
  when: iptables_status.stat.exists
  
- name: check if firewalld exists
  stat: path=/usr/lib/systemd/system/firewalld.service
  register: firewalld_status

- name: enable firewalld
  systemd:
    name: firewalld
    state: started
    enabled: yes
  when: firewalld_status.stat.exists
  register: firewall_started

- name: activate public zone by adding interface to public zone
  firewalld:
    zone: public
    interface: "{{ ansible_default_ipv4.interface }}"
    state: enabled
    permanent: yes
    immediate: yes
  when: firewall_started is success
    
- name: remove dhcpv6-client service from zone public
  firewalld:
    zone: public
    service: dhcpv6-client
    state: disabled
    permanent: yes
    immediate: yes
  when: firewall_started is success

- name: add database listen port to zone public
  firewalld:
    zone: public
    port: 1521/tcp
    state: enabled
    permanent: yes
    immediate: yes
  when: firewall_started is success

# configure selinux
- name: enable selinux
  selinux:
    policy=targeted
    state=permissive
  notify:
    - restart server
    - wait for server to restart


# /etc/ansible/roles/database_host/handlers/main.yml

- name: restart server
  shell: sleep 2 && shutdown -r now "Ansible updates triggered"
  async: 1
  poll: 0
  ignore_errors: True
  
- name: wait for server to restart
  local_action: wait_for
    host={{ inventory_hostname }}
    port=22
    state=started
    delay=10
    timeout=300
  become: false

Installation

To install Oracle Database software, we use the following:

Used variables:

##### database/grid version specification #####
software_version: "19.0.0"
# Directory where the software to be installed is located
software_directory: "/mnt/hgfs/vmware_temp/database_{{ software_version }}"

# when software version is 18 or higher choose image or rpm based install
image_based_software_installation: "true"
image_name: "V982063-01.zip"

rpm_based_software_installation: "false"
rpm_path: "oracle-database-ee-19c-1.0-1.x86_64.rpm"

##### database directories #####
# The scripts create files that are placed in this directory
temporary_directory: "/u01/repsonse_files_{{ software_version }}"
# root directory
root_directory: "/u01"
# Scripts directory
scripts_directory: "{{ root_directory }}/app/scripts"
# Location of the Oracle inventory
oracle_inventory_home: "{{ root_directory }}/app/oraInventory"
# Base directory
base_directory: "{{ root_directory }}/app/oracle"
# Oracle home
oracle_home: "{{ base_directory }}/product/{{ software_version }}/dbhome"

##### install options #####
oracle_hostname: "set_in_environment_specific_group_vars_file"
oracle_install_option: "INSTALL_DB_SWONLY"
oracle_installation_edition: "EE"
oracle_execute_root_script: "false"

##### files #####
run_installer_12c: "{{ software_directory }}/runInstaller"
run_installer: "{{ oracle_home }}/runInstaller"
net_creation_assistent: "{{ oracle_home }}/bin/netca"
database_creation_assistent: "{{ oracle_home }}/bin/dbca"

silent_install_file: "{{ temporary_directory }}/silent-install.rsp"
silent_listener_config_file: "{{ temporary_directory }}/silent-listener-config.rsp"
silent_database_config_file: "{{ temporary_directory }}/silent-database-config.rsp"

root_sh: "{{ oracle_home }}/root.sh"
orainst_root_sh: "{{ oracle_inventory_home }}/orainstRoot.sh"

orainst_loc: "/etc/oraInst.loc"
oratab: "/etc/oratab"

listener_ora: "{{ oracle_home }}/network/admin/listener.ora"
sp_file: "{{ oracle_home }}/dbs/spfile{{ database_sid }}.ora"
database_service: "{{ scripts_directory }}/database_service"

Create directory structure and set the permissions.
Based on the image_based_software_installation and rpm_based_software_installation parameters, the software is installed by using either an image or an RPM (or when both are false the old fashioned 12c way). Note that
Starting with Oracle Database 18c, installation and configuration of Oracle Database software is simplified with image-based installation. To install Oracle Database, create the new Oracle home, extract the image file into the newly-created Oracle home, and run the setup wizard to register the Oracle Database product. Using image-based installation, you can install and upgrade Oracle Database for single-instance and cluster configurations. This installation feature streamlines the installation process and supports automation of large-scale custom deployments. You can also use this installation method for deployment of customized images, after you patch the base-release software with the necessary Release Updates (Updates) or Release Update Revisions (Revisions).

Starting with Oracle Database 18c, you can install a single-instance Oracle Database or an Oracle Database Instant Client software using RPM packages. An RPM-based installation performs preinstallation checks, extracts the database software, reassigns ownership of the extracted software to the preconfigured user and groups, maintains the Oracle inventory, and executes all root operations required to configure the Oracle Database software for a single-instance Oracle Database creation and configuration. The RPM–based installation process detects when the minimum requirements for an installation are not met and prompts you to finish these minimum preinstallation requirements. An RPM-based installation performs a software-only Oracle Database installation and creates an Oracle home. After the Oracle home is created, you can then use Oracle Database Configuration Assistant (Oracle DBCA) to create an Oracle Database.

Create response file, for example,

####################################################################
## Copyright(c) Oracle Corporation 1998,2019. All rights reserved.##
##                                                                ##
## Specify values for the variables listed below to customize     ##
## your installation.                                             ##
##                                                                ##
## Each variable is associated with a comment. The comment        ##
## can help to populate the variables with the appropriate        ##
## values.                                                        ##
##                                                                ##
## IMPORTANT NOTE: This file contains plain text passwords and    ##
## should be secured to have read permission only by oracle user  ##
## or db administrator who owns this installation.                ##
##                                                                ##
####################################################################


#------------------------------------------------------------------------------
# Do not change the following system generated value. 
#------------------------------------------------------------------------------
oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v19.0.0

#-------------------------------------------------------------------------------
# Specify the installation option.
# It can be one of the following:
#   - INSTALL_DB_SWONLY
#   - INSTALL_DB_AND_CONFIG
#-------------------------------------------------------------------------------
oracle.install.option={{ oracle_install_option }}

#-------------------------------------------------------------------------------
# Specify the Unix group to be set for the inventory directory.  
#-------------------------------------------------------------------------------
UNIX_GROUP_NAME={{ oracle_install_group }}

#-------------------------------------------------------------------------------
# Specify the location which holds the inventory files.
# This is an optional parameter if installing on
# Windows based Operating System.
#-------------------------------------------------------------------------------
INVENTORY_LOCATION={{ oracle_inventory_home }}
#-------------------------------------------------------------------------------
# Specify the complete path of the Oracle Home. 
#-------------------------------------------------------------------------------
ORACLE_HOME={{ oracle_home }}

#-------------------------------------------------------------------------------
# Specify the complete path of the Oracle Base. 
#-------------------------------------------------------------------------------
ORACLE_BASE={{ base_directory }}
...

Run runInstaller.

Run orainstRoot.sh and root.sh (see here for more information). Note that

Starting with Oracle Database 19c, the database installer, or setup wizard, provides options to set up permissions to run the root configuration scripts automatically, as required, during a database installation. You continue to have the option to run the root configuration scripts manually. Setting up permissions for root configuration scripts to run without user intervention can simplify database installation and help avoid inadvertent permission errors.

As we are using Ansible the root script execution mode

################################################################################
#                                                                              #
#                      Root script execution configuration                     #
#                                                                              #
################################################################################

#-------------------------------------------------------------------------------------------------------
# Specify the root script execution mode.
#
#   - true  : To execute the root script automatically by using the appropriate configuration methods.
#   - false : To execute the root script manually.
#
# If this option is selected, password should be specified on the console.
#-------------------------------------------------------------------------------------------------------
oracle.install.db.rootconfig.executeRootScript={{ oracle_execute_root_script }}

#--------------------------------------------------------------------------------------
# Specify the configuration method to be used for automatic root script execution.
#
# Following are the possible choices:
#   - ROOT
#   - SUDO
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.configMethod=
#--------------------------------------------------------------------------------------
# Specify the absolute path of the sudo program.
#
# Applicable only when SUDO configuration method was chosen.
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.sudoPath=

#--------------------------------------------------------------------------------------
# Specify the name of the user who is in the sudoers list. 
# Applicable only when SUDO configuration method was chosen.
# Note:For Single Instance database installations,the sudo user name must be the username of the user installing the database.
#--------------------------------------------------------------------------------------
oracle.install.db.rootconfig.sudoUserName=

is set to false.

To combine the above in an Ansible playbook, we have the following

# /etc/ansible/roles/oracle_database_install/tasks/main.yml

- name: create root directory
  file:
    path: "{{ root_directory }}"
    state: directory
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
    mode: 0755
    
- name: create directories
  file:
    path: "{{ item }}"
    state: directory
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
    mode: 0755
  when: not( (item is undefined) or (item is none) or (item | trim=='') )
  with_items:
    - "{{ temporary_directory }}"
    - "{{ scripts_directory }}"
    - "{{ oracle_home }}"

- name: image based software installation
  unarchive:
    copy: no
    creates: "{{ oracle_home }}/runInstaller"
    src: "{{ software_directory }}/{{ image_name }}"
    dest: "{{ oracle_home }}"
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
  when: image_based_software_installation
    
- name: rpm based software installation
  yum:
    name: "{{ software_directory }}/{{ rpm_name }}"
    state: present
  when: rpm_based_software_installation
    
- name: create silent install file
  template:
    src: roles/oracle_database_install/templates/db_install_rsp_{{ software_version }}.j2 
    dest: "{{ silent_install_file }}"
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
    mode: 0644
    
- name: check if orainstroot exists
  stat: path="{{ orainst_root_sh }}"
  register: orainstroot_status
    
- name: 12c oracle database install
  become: true
  become_user: "{{ oracle_install_user }}"
  become_method: sudo
  command: /bin/sh -c "{{ software_directory }}/runInstaller -silent -ignorePrereq -waitforcompletion -responseFile {{ silent_install_file }}"
  when: not orainstroot_status.stat.exists and not image_based_software_installation and not rpm_based_software_installation
  ignore_errors: True
  register: oracle_database_installed
  
- name: debug 12c oracle database install
  debug:
    var=oracle_database_installed
  when: oracle_database_installed is changed

- name: install oracle database
  become: true
  become_user: "{{ oracle_install_user }}"
  become_method: sudo
  command: /bin/sh -c "{{ oracle_home }}/runInstaller -silent -ignorePrereq -waitforcompletion -responseFile {{ silent_install_file }}"
  when: not orainstroot_status.stat.exists and (image_based_software_installation or rpm_based_software_installation)
  ignore_errors: True
  register: oracle_database_installed
  
- name: debug install oracle database
  debug:
    var=oracle_database_installed
  when: oracle_database_installed is changed

- name: run orainstRoot.sh
  command: /bin/sh "{{ orainst_root_sh }}"
    creates="{{ orainst_loc }}"

- name: run root.sh
  command: /bin/sh "{{ root_sh }}"
    creates="{{ oratab }}"
    
- name: delete temporary setup files that contain usernames and passwords
  file:
    path: "{{ item }}"
    state: absent
  loop:
    - "{{ silent_install_file }}"

Configuration

To configure the Oracle Database, we use the following:

Used variables

##### configuration options #####
listener_name: "LISTENER"
listener_protocol: "TCP;1521"

global_database_name: "orcl.ms.com"
database_sid: "orcl"
database_config_type: "SI"
policy_managed: "FALSE"
create_server_pool: "FALSE"
create_as_container_database: "FALSE"
template_name: "General_Purpose.dbc"
sys_password: "set_in_environment_specific_group_vars_file"
system_password: "set_in_environment_specific_group_vars_file"
em_configuration: "NONE"
data_vault_configuration: "FALSE"
oracle_label_security_configuration: "FALSE"
storage_type: "FS"
character_set: "AL32UTF8"
register_with_dir_service: "FALSE"
sample_schema: "FALSE"
memory_percentage: 40
database_type: "OLTP"
automatic_memory_management: "TRUE"

Create a response file for the net configuration assistent, for example,

###################################################################### 
## Copyright(c) 1998, 2018 Oracle Corporation. All rights reserved. ## 
##                                                                  ## 
## Specify values for the variables listed below to customize your  ## 
## installation.                                                    ## 
##                                                                  ## 
## Each variable is associated with a comment. The comment          ## 
## identifies the variable type.                                    ## 
##                                                                  ## 
## Please specify the values in the following format:               ## 
##                                                                  ## 
##         Type         Example                                     ## 
##         String       "Sample Value"                              ## 
##         Boolean      True or False                               ## 
##         Number       1000                                        ## 
##         StringList   {"String value 1","String Value 2"}         ## 
##                                                                  ## 
######################################################################
##                                                                  ## 
## This sample response file causes the Oracle Net Configuration    ##
## Assistant (NetCA) to complete an Oracle Net configuration during ##
## a custom install of the Oracle12c server which is similar to     ##
## what would be created by the NetCA during typical Oracle12c      ##
## install. It also documents all of the NetCA response file        ##
## variables so you can create your own response file to configure  ##
## Oracle Net during an install the way you wish.                   ##
##                                                                  ## 
###################################################################### 

[GENERAL]
RESPONSEFILE_VERSION="19.0"
CREATE_TYPE="CUSTOM"

#-------------------------------------------------------------------------------
# Name       : SHOW_GUI
# Datatype   : Boolean
# Description: This variable controls appearance/suppression of the NetCA GUI,
# Pre-req    : N/A
# Default    : TRUE
# Note:
# This must be set to false in order to run NetCA in silent mode. 
# This is a substitute of "/silent" flag in the NetCA command line.
# The command line flag has precedence over the one in this response file.
# This feature is present since 10.1.0.3.
#-------------------------------------------------------------------------------
#SHOW_GUI=false

#-------------------------------------------------------------------------------
# Name       : LOG_FILE
# Datatype   : String
# Description: If present, NetCA will log output to this file in addition to the
#        standard out.
# Pre-req    : N/A
# Default    : NONE
# Note:
#  This is a substitute of "/log" in the NetCA command line.
# The command line argument has precedence over the one in this response file.
# This feature is present since 10.1.0.3.
#-------------------------------------------------------------------------------
#LOG_FILE=""$ORACLE_BASE/cfgtoollogs/netca/netca.log""

[oracle.net.ca]
#INSTALLED_COMPONENTS;StringList;list of installed components
# The possible values for installed components are:
# "net8","server","client","aso", "cman", "javavm" 
INSTALLED_COMPONENTS={"server","net8","javavm"}

#INSTALL_TYPE;String;type of install
# The possible values for install type are:
# "typical","minimal" or "custom"
INSTALL_TYPE=""typical""

#LISTENER_NUMBER;Number;Number of Listeners
# A typical install sets one listener 
LISTENER_NUMBER=1

#LISTENER_NAMES;StringList;list of listener names
# The values for listener are:
# "LISTENER","LISTENER1","LISTENER2","LISTENER3", ...
# A typical install sets only "LISTENER" 
LISTENER_NAMES={"{{ listener_name }}"}

#LISTENER_PROTOCOLS;StringList;list of listener addresses (protocols and parameters separated by semicolons)
# The possible values for listener protocols are:
# "TCP;1521","TCPS;2484","NMP;ORAPIPE","IPC;IPCKEY","VI;1521" 
# For multiple listeners, separate them with commas ex "TCP;1521","TCPS;2484"
# For multiple protocols in single listener, separate them with "&" ex  "TCP;1521&TCPS;2484"
# A typical install sets only "TCP;1521" 
LISTENER_PROTOCOLS={"{{ listener_protocol }}"}

#LISTENER_START;String;name of the listener to start, in double quotes
LISTENER_START=""{{ listener_name }}""

#NAMING_METHODS;StringList;list of naming methods
# The possible values for naming methods are: 
# LDAP, TNSNAMES, ONAMES, HOSTNAME, NOVELL, NIS, DCE
# A typical install sets only: "TNSNAMES","ONAMES","HOSTNAMES" 
# or "LDAP","TNSNAMES","ONAMES","HOSTNAMES" for LDAP
NAMING_METHODS={"TNSNAMES","ONAMES","HOSTNAME"}
...

Run netca (more information on the net configuration assistent can be found here).

Create a response file for the database configuration assistent, for example,

##############################################################################
##                                                                          ##
##                            DBCA response file                            ##
##                            ------------------                            ##
## Copyright(c) Oracle Corporation 1998,2019. All rights reserved.          ##
##                                                                          ##
## Specify values for the variables listed below to customize               ##
## your installation.                                                       ##
##                                                                          ##
## Each variable is associated with a comment. The comment                  ##
## can help to populate the variables with the appropriate                  ##
## values.                                                                  ##
##                                                                          ##
## IMPORTANT NOTE: This file contains plain text passwords and              ##
## should be secured to have read permission only by oracle user            ##
## or db administrator who owns this installation.                          ##
##############################################################################
#-------------------------------------------------------------------------------
# Do not change the following system generated value. 
#-------------------------------------------------------------------------------
responseFileVersion=/oracle/assistants/rspfmt_dbca_response_schema_v19.0.0

#-----------------------------------------------------------------------------
# Name          : gdbName
# Datatype      : String
# Description   : Global database name of the database
# Valid values  : <db_name>.<db_domain> - when database domain isn't NULL
#                 <db_name>             - when database domain is NULL
# Default value : None
# Mandatory     : Yes
#-----------------------------------------------------------------------------
gdbName={{ global_database_name }}

#-----------------------------------------------------------------------------
# Name          : sid
# Datatype      : String
# Description   : System identifier (SID) of the database
# Valid values  : Check Oracle19c Administrator's Guide
# Default value : <db_name> specified in GDBNAME
# Mandatory     : No
#-----------------------------------------------------------------------------
sid={{ database_sid }}

#-----------------------------------------------------------------------------
# Name          : databaseConfigType
# Datatype      : String
# Description   : database conf type as Single Instance, Real Application Cluster or Real Application Cluster One Nodes database
# Valid values  : SI\RAC\RACONENODE
# Default value : SI
# Mandatory     : No
#-----------------------------------------------------------------------------
databaseConfigType={{ database_config_type }}
...

Run dbca (more information on the database configuration assistent can be found here).

To combine the above in an Ansible playbook, we have the following

# /etc/ansible/roles/oracle_database_config/tasks/main.yml

- name: create silent listener config file
  template:
    src: roles/oracle_database_config/templates/netca_rsp_{{ software_version }}.j2 
    dest: "{{ silent_listener_config_file }}"
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
    mode: 0644
  register: silent_listener_config_file_created

- name: configure listener
  become: true
  become_user: "{{ oracle_install_user }}"
  become_method: sudo
  command: /bin/sh -c "{{ net_creation_assistent }} -silent -responsefile {{ silent_listener_config_file }}"
    creates="{{ listener_ora }}"
  when: silent_listener_config_file_created is success
  register: listener_configured
  
- name: debug configure listener
  debug:
    var=listener_configured
  when: listener_configured is changed

- name: create silent database config file
  template:
    src: roles/oracle_database_config/templates/dbca_rsp_{{ software_version }}.j2 
    dest: "{{ silent_database_config_file }}"
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
    mode: 0644
  register: silent_database_config_file_created

- name: configure oracle database
  become: true
  become_user: "{{ oracle_install_user }}"
  become_method: sudo
  command: /bin/sh -c "{{ database_creation_assistent }} -silent -createDatabase -responseFile {{ silent_database_config_file }}"
    creates="{{ sp_file }}"
  when: silent_database_config_file_created is success
  register: oracle_database_configured
  
- name: debug configure oracle database
  debug:
    var=oracle_database_configured
  when: oracle_database_configured is changed
    
- name: create has script
  template:
    src: roles/oracle_database_config/templates/has_service.j2
    dest: "{{ has_service }}"
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
    mode: 0744
    
- name: create database script
  template:
    src: roles/oracle_database_config/templates/database_service.j2 
    dest: "{{ database_service }}"
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
    mode: 0744
    
- name: copy scripts
  copy: 
    src: roles/oracle_database_config/files/oracle/lifecycle/
    dest: "{{ scripts_directory }}"
    owner: "{{ oracle_install_user }}"
    group: "{{ oracle_install_group }}"
    mode: 0744

- name: delete temporary setup files that contain usernames and passwords
  file:
    path: "{{ item }}"
    state: absent
  loop:
    - "{{ silent_listener_config_file }}"
    - "{{ silent_database_config_file }}"

Run Ansible

A good way to keep production environments and other environments separate is to use separate inventory files. In our example the inventory file for test has the following contents

# /etc/ansible/inventories/test/hosts.cfg

[database_host]
datanode

[database_hosts:children]
database_host

[all:children]
database_hosts

Note that we use group_vars to override variables, for example,

##### User and group settings #####
oracle_install_password:         "..."

##### install options #####
oracle_hostname:                 "{{ inventory_hostname  }}"

##### configuration options #####
sys_password:                    "..."
system_password:                 "..."

Note that this file controls the differences between environments, such as address settings. This file also contains sensitive data, such as passwords, and as a result this file should be encypted. We can encrypt files by using Vault. Vault is a feature of Ansible that allows keeping sensitive data such as passwords or keys in encrypted files, rather than as plaintext in roles. These vault files can then be distributed or placed in source control. To enable this feature, a command line tool, ansible-vault is used to edit files, and a command line flag --ask-vault-pass or --vault-password-file is used. Alternately, we may specify the location of a password file or command Ansible to always prompt for the password in the ansible.cfg file (vault_password_file = /root/.rommel). This option requires no command line flag usage. To encrypt the file, we can use ansible-vault encrypt filename.yml.

The playbook that uses the roles defined above, has the following contents

##########
# database host configuration
# ansible-playbook oracle_database.yml --inventory=inventories/test/hosts.cfg --tags "host"
#
# database installation
# ansible-playbook oracle_database.yml --inventory=inventories/test/hosts.cfg --tags "install"
#
# database configuration
# ansible-playbook oracle_database.yml --inventory=inventories/test/hosts.cfg --tags "config"

- hosts: database_hosts
  user: ansible
  become: true
  become_user: root
  become_method: sudo
  
  roles:
    - { role: oracle_database_vars, tags: ['host','install','config'] }
    - { role: oracle_database_host, tags: ['host'] }
    - { role: oracle_database_install, tags: ['install'] }
    - { role: oracle_database_config, tags: ['config'] }

in which, we have also defined tags.

To configure the host, we use

[ansible@datanode ansible]$ ansible-playbook oracle_database.yml --inventory=inventories/test/hosts.cfg --tags "host"

PLAY [database_hosts] ******************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [datanode]

TASK [oracle_database_host : install necessary packages] *******************************************************************************************************************************************************
ok: [datanode] => (item=binutils)
changed: [datanode] => (item=compat-libcap1)
changed: [datanode] => (item=compat-libstdc++-33)
ok: [datanode] => (item=dos2unix)
ok: [datanode] => (item=gcc)
changed: [datanode] => (item=gcc-c++)
ok: [datanode] => (item=glibc)
ok: [datanode] => (item=glibc-devel)
ok: [datanode] => (item=kmod)
changed: [datanode] => (item=ksh)
ok: [datanode] => (item=libaio)
changed: [datanode] => (item=libaio-devel)
ok: [datanode] => (item=libgcc)
ok: [datanode] => (item=libstdc++)
ok: [datanode] => (item=libstdc++-devel)
changed: [datanode] => (item=libX11)
ok: [datanode] => (item=libXau)
ok: [datanode] => (item=libxcb)
changed: [datanode] => (item=libXext)
changed: [datanode] => (item=libXi)
changed: [datanode] => (item=libXrender)
changed: [datanode] => (item=libXtst)
changed: [datanode] => (item=lsof)
ok: [datanode] => (item=ksh)
ok: [datanode] => (item=make)
ok: [datanode] => (item=net-tools)
ok: [datanode] => (item=perl)
changed: [datanode] => (item=smartmontools)
changed: [datanode] => (item=strace)
changed: [datanode] => (item=sysstat)
changed: [datanode] => (item=unzip)

TASK [oracle_database_host : create group] *********************************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_host : create user] **********************************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_host : set sysctl /etc/sysctl.d/10-default_sysctl.conf] **********************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_host : run sysctl -p] ********************************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_host : set /etc/security/limits.conf] ****************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_host : check if iptables exists] *********************************************************************************************************************************************************
ok: [datanode]

TASK [oracle_database_host : disable iptables] *****************************************************************************************************************************************************************
skipping: [datanode]

TASK [oracle_database_host : check if firewalld exists] ********************************************************************************************************************************************************
ok: [datanode]

TASK [oracle_database_host : enable firewalld] *****************************************************************************************************************************************************************
ok: [datanode]

TASK [oracle_database_host : activate public zone by adding interface to public zone] **************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_host : remove dhcpv6-client service from zone public] ************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_host : add database listen port to zone public] ******************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_host : enable selinux] *******************************************************************************************************************************************************************
changed: [datanode]

RUNNING HANDLER [oracle_database_host : restart server] ********************************************************************************************************************************************************
changed: [datanode]

RUNNING HANDLER [oracle_database_host : wait for server to restart] ********************************************************************************************************************************************
ok: [datanode -> localhost]

PLAY RECAP *****************************************************************************************************************************************************************************************************
datanode                       : ok=16   changed=11   unreachable=0    failed=0

To install the database software, we use

[ansible@datanode ansible]$ ansible-playbook oracle_database.yml --inventory=inventories/test/hosts.cfg --tags "install"

PLAY [database_hosts] ******************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [datanode]

TASK [oracle_database_install : create root directory] *********************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_install : create directories] ************************************************************************************************************************************************************
changed: [datanode] => (item=/u01/repsonse_files_19.0.0)
changed: [datanode] => (item=/u01/app/scripts)
changed: [datanode] => (item=/u01/app/oracle/product/19.0.0/dbhome)

TASK [oracle_database_install : image based software installation] *********************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_install : rpm based software installation] ***********************************************************************************************************************************************
skipping: [datanode]

TASK [oracle_database_install : create silent install file] ****************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_install : 12c oracle database install] ***************************************************************************************************************************************************
skipping: [datanode]

TASK [oracle_database_install : debug 12c oracle database install] *********************************************************************************************************************************************
skipping: [datanode]

TASK [oracle_database_install : install oracle database] *******************************************************************************************************************************************************
[WARNING]: Module remote_tmp /home/oracle/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with
the correct permissions manually
changed: [datanode]

TASK [oracle_database_install : debug install oracle database] *************************************************************************************************************************************************
ok: [datanode] => {
    "oracle_database_installed": {
        "changed": true,
        "cmd": [
            "/bin/sh",
            "-c",
            "/u01/app/oracle/product/19.0.0/dbhome/runInstaller -silent -ignorePrereq -waitforcompletion -responseFile /u01/repsonse_files_19.0.0/silent-install.rsp"
        ],
        "delta": "0:01:02.026905",
        "end": "2019-05-08 11:27:39.449180",
        "failed": false,
        "rc": 0,
        "start": "2019-05-08 11:26:37.422275",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "Launching Oracle Database Setup Wizard...\n\nThe response file for this session can be found at:\n /u01/app/oracle/product/19.0.0/dbhome/install/response/db_2019-05-08_11-26-37AM.rsp\n\nYou can find the log of this install session at:\n /tmp/InstallActions2019-05-08_11-26-37AM/installActions2019-05-08_11-26-37AM.log\n\nAs a root user, execute the following script(s):\n\t1. /u01/app/oraInventory/orainstRoot.sh\n\t2. /u01/app/oracle/product/19.0.0/dbhome/root.sh\n\nExecute /u01/app/oraInventory/orainstRoot.sh on the following nodes: \n[datanode]\nExecute /u01/app/oracle/product/19.0.0/dbhome/root.sh on the following nodes: \n[datanode]\n\n\nSuccessfully Setup Software.\nMoved the install session logs to:\n /u01/app/oraInventory/logs/InstallActions2019-05-08_11-26-37AM",
        "stdout_lines": [
            "Launching Oracle Database Setup Wizard...",
            "",
            "The response file for this session can be found at:",
            " /u01/app/oracle/product/19.0.0/dbhome/install/response/db_2019-05-08_11-26-37AM.rsp",
            "",
            "You can find the log of this install session at:",
            " /tmp/InstallActions2019-05-08_11-26-37AM/installActions2019-05-08_11-26-37AM.log",
            "",
            "As a root user, execute the following script(s):",
            "\t1. /u01/app/oraInventory/orainstRoot.sh",
            "\t2. /u01/app/oracle/product/19.0.0/dbhome/root.sh",
            "",
            "Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes: ",
            "[datanode]",
            "Execute /u01/app/oracle/product/19.0.0/dbhome/root.sh on the following nodes: ",
            "[datanode]",
            "",
            "",
            "Successfully Setup Software.",
            "Moved the install session logs to:",
            " /u01/app/oraInventory/logs/InstallActions2019-05-08_11-26-37AM"
        ]
    }
}

TASK [oracle_database_install : run orainstRoot.sh] ************************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_install : run root.sh] *******************************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_install : delete temporary setup files that contain usernames and passwords] *************************************************************************************************************
changed: [datanode] => (item=/u01/repsonse_files_19.0.0/silent-install.rsp)

PLAY RECAP *****************************************************************************************************************************************************************************************************
datanode                       : ok=10    changed=10    unreachable=0    failed=0

To configure the database, we use

[ansible@datanode ansible]$ ansible-playbook oracle_database.yml --inventory=inventories/test/hosts.cfg --tags "config"

PLAY [database_hosts] ******************************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************************
ok: [datanode]

TASK [oracle_database_config : create silent listener config file] *********************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_config : configure listener] *************************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_config : debug configure listener] *******************************************************************************************************************************************************
ok: [datanode] => {
    "listener_configured": {
        "changed": true,
        "cmd": [
            "/bin/sh",
            "-c",
            "/u01/app/oracle/product/19.0.0/dbhome/bin/netca -silent -responsefile /u01/repsonse_files_19.0.0/silent-listener-config.rsp"
        ],
        "delta": "0:00:01.370024",
        "end": "2019-05-08 11:31:38.365648",
        "failed": false,
        "rc": 0,
        "start": "2019-05-08 11:31:36.995624",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "\nParsing command line arguments:\n    Parameter \"silent\" = true\n    Parameter \"responsefile\" = /u01/repsonse_files_19.0.0/silent-listener-config.rsp\nDone parsing command line arguments.\nOracle Net Services Configuration:\nProfile configuration complete.\nOracle Net Listener Startup:\n    Running Listener Control: \n      /u01/app/oracle/product/19.0.0/dbhome/bin/lsnrctl start LISTENER\n    Listener Control complete.\n    Listener started successfully.\nListener configuration complete.\nOracle Net Services configuration successful. The exit code is 0",
        "stdout_lines": [
            "",
            "Parsing command line arguments:",
            "    Parameter \"silent\" = true",
            "    Parameter \"responsefile\" = /u01/repsonse_files_19.0.0/silent-listener-config.rsp",
            "Done parsing command line arguments.",
            "Oracle Net Services Configuration:",
            "Profile configuration complete.",
            "Oracle Net Listener Startup:",
            "    Running Listener Control: ",
            "      /u01/app/oracle/product/19.0.0/dbhome/bin/lsnrctl start LISTENER",
            "    Listener Control complete.",
            "    Listener started successfully.",
            "Listener configuration complete.",
            "Oracle Net Services configuration successful. The exit code is 0"
        ]
    }
}

TASK [oracle_database_config : create silent database config file] *********************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_config : configure oracle database] ******************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_config : debug configure oracle database] ************************************************************************************************************************************************
ok: [datanode] => {
    "oracle_database_configured": {
        "changed": true,
        "cmd": [
            "/bin/sh",
            "-c",
            "/u01/app/oracle/product/19.0.0/dbhome/bin/dbca -silent -createDatabase -responseFile /u01/repsonse_files_19.0.0/silent-database-config.rsp"
        ],
        "delta": "0:08:13.818718",
        "end": "2019-05-08 11:39:52.847777",
        "failed": false,
        "rc": 0,
        "start": "2019-05-08 11:31:39.029059",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "Prepare for db operation\n10% complete\nCopying database files\n40% complete\nCreating and starting Oracle instance\n42% complete\n46% complete\n50% complete\n54% complete\n60% complete\nCompleting Database Creation\n66% complete\n69% complete\n70% complete\nExecuting Post Configuration Actions\n100% complete\nDatabase creation complete. For details check the logfiles at:\n /u01/app/oracle/cfgtoollogs/dbca/ORCL.\nDatabase Information:\nGlobal Database Name:ORCL\nSystem Identifier(SID):ORCL\nLook at the log file \"/u01/app/oracle/cfgtoollogs/dbca/ORCL/ORCL.log\" for further details.",
        "stdout_lines": [
            "Prepare for db operation",
            "10% complete",
            "Copying database files",
            "40% complete",
            "Creating and starting Oracle instance",
            "42% complete",
            "46% complete",
            "50% complete",
            "54% complete",
            "60% complete",
            "Completing Database Creation",
            "66% complete",
            "69% complete",
            "70% complete",
            "Executing Post Configuration Actions",
            "100% complete",
            "Database creation complete. For details check the logfiles at:",
            " /u01/app/oracle/cfgtoollogs/dbca/ORCL.",
            "Database Information:",
            "Global Database Name:ORCL",
            "System Identifier(SID):ORCL",
            "Look at the log file \"/u01/app/oracle/cfgtoollogs/dbca/ORCL/ORCL.log\" for further details."
        ]
    }
}

TASK [oracle_database_config : create has script] **************************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_config : create database script] *********************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_config : copy scripts] *******************************************************************************************************************************************************************
changed: [datanode]

TASK [oracle_database_config : delete temporary setup files that contain usernames and passwords] **************************************************************************************************************
changed: [datanode] => (item=/u01/repsonse_files_19.0.0/silent-listener-config.rsp)
changed: [datanode] => (item=/u01/repsonse_files_19.0.0/silent-database-config.rsp)

PLAY RECAP *****************************************************************************************************************************************************************************************************
datanode                       : ok=11   changed=8    unreachable=0    failed=0

with the result (a running database)

[oracle@datanode scripts]$ ./database_service status
The Oracle base has been set to /u01/app/oracle
Checking Database

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 08-MAY-2019 13:12:49

Copyright (c) 1991, 2019, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=datanode.ms.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                08-MAY-2019 11:46:32
Uptime                    0 days 1 hr. 26 min. 16 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/19.0.0/dbhome/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/datanode/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=datanode.ms.com)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "ORCL" has 1 instance(s).
  Instance "ORCL", status READY, has 1 handler(s) for this service...
Service "ORCLXDB" has 1 instance(s).
  Instance "ORCL", status READY, has 1 handler(s) for this service...
The command completed successfully

To test the set-up, we create a data source in WebLogic, i.e.,

[weblogic@javanode1 scripts]$ cat /u01/app/oracle/weblogic12.2.1/configuration/domains/tryout_domain/config/jdbc/data_source-jdbc.xml
<jdbc-data-source ...>
  <name>data_source</name>
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@//datanode.ms.com:1521/orcl.ms.com</url>
    <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name>
    <properties>
      <property>
        <name>user</name>
        <value>example</value>
      </property>
    </properties>
    <password-encrypted>{AES}iJxm6loPE9PHASnV4wMS4xkje9JxdNj/u1Vp9sW5lp0=</password-encrypted>
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <initial-capacity>1</initial-capacity>
    <max-capacity>15</max-capacity>
    <min-capacity>1</min-capacity>
    <connection-creation-retry-frequency-seconds>30</connection-creation-retry-frequency-seconds>
    <test-table-name>SQL ISVALID</test-table-name>
    <statement-cache-size>0</statement-cache-size>
    <statement-cache-type>LRU</statement-cache-type>
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>jdbc/exampleDS</jndi-name>
    <global-transactions-protocol>TwoPhaseCommit</global-transactions-protocol>
  </jdbc-data-source-params>
</jdbc-data-source>

and use the application presented here to issue some requests

[weblogic@javanode1 scripts]$ cat /u01/app/oracle/weblogic12.2.1/configuration/domains/tryout_domain/servers/application_server_1/logs/application_server_1.out
...
RECEIVED OBJECT MESSAGE 145vbizoe7myg 5418
RECEIVED OBJECT MESSAGE fb6bol1924w4 708
RECEIVED OBJECT MESSAGE 1ch63dy8yqai9 2187
RECEIVED OBJECT MESSAGE 1g7gaw57fzotn 1331
RECEIVED OBJECT MESSAGE oguc0qagbp4u 9137
RECEIVED OBJECT MESSAGE bqvwlsb9zv89 7516
RECEIVED OBJECT MESSAGE dx4mjshkh1vk 7154
RECEIVED OBJECT MESSAGE 1oiykwu7nzfhk 7478
RECEIVED OBJECT MESSAGE 1wkdoab0estec 9915
RECEIVED OBJECT MESSAGE 19hi8q8q6x7yu 9398
RECEIVED OBJECT MESSAGE 1cbh6wevr0tpx 8587
RECEIVED OBJECT MESSAGE 1iw33pa5jzm1y 9239
RECEIVED OBJECT MESSAGE 1fgn7a86yvn3k 1431
RECEIVED OBJECT MESSAGE 1sxe62ll5vyz3 976
RECEIVED OBJECT MESSAGE uycy5zje69yt 5053
RECEIVED OBJECT MESSAGE 1vu33l45a4nv1 9714
RECEIVED OBJECT MESSAGE henvcnmrykhx 2779
RECEIVED OBJECT MESSAGE 1sysqs60tsjka 8564
RECEIVED OBJECT MESSAGE 162nwi88unxav 5078
RECEIVED OBJECT MESSAGE 1bev9nc1b2t6a 778

[oracle@datanode scripts]$ . /usr/local/bin/oraenv
ORACLE_SID = [oracle] ? orcl
The Oracle base has been set to /u01/app/oracle
[oracle@datanode scripts]$ sqlplus

SQL*Plus: Release 19.0.0.0.0 - Production on Mon May 20 13:27:52 2019
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Enter user-name: example
Enter password:
Last Successful login time: Mon May 20 2019 13:23:37 +02:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> describe persoon;
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 SOFINUMMER                                NOT NULL NUMBER(9)
 NAAM                                      NOT NULL VARCHAR2(50)
 FAMILYSOFINUMMER                                   NUMBER(9)

SQL> select sofinummer from persoon where naam like '1bev9nc1b2t6a';

SOFINUMMER
----------
       778

Nah. No one will build a robot that does nothing but drink coffee - Wally.

References

[1] Ansible Documentation.
[2] Oracle Database Documentation.

A Little Bit of Everything

2019-04-07T14:44:00.001+02:00

In the following we give an example of how to set-up various platform components, such as,

A Firewall.
An SSL Offloader and Proxy.
A Load Balancer.
A JavaEE Application Server.
A Database or another Database.
A Directory Server.
An Application & Infrastructure Monitoring tool.

In the example set-up, we use the following hosts

[testuser@db1 ~]$ cat /etc/hosts
127.0.0.1         localhost
172.20.40.141     ansible.ms.com     ansible
172.20.40.142     dmz.ms.com         dmz
172.20.40.143     proxy.ms.com       proxy
172.20.40.144     java1.ms.com       java1
172.20.40.145     db1.ms.com         db1
172.20.40.146     java2.ms.com       java2
172.20.40.151     test151.ms.com     test151

An example architecture could be as follows

The Ansible node runs as the name suggests our automation tool, in this case Ansible.
The DMZ node runs the (main) Firewall, the SSL Offloader, and the (main) Proxy.
The Proxy node runs the Load Balancer.
The Java nodes run a JavaEE Application Server cluster.
The DB node runs the Databases and the Directory Server.
The test node runs the Application & Infrastructure Monitoring tool.

Before, we begin we define the Ansible Inventory:

[dmz_host]
dmz

[proxy_host]
proxy

[java1_host]
java1

[java2_host]
java2

[db1_host]
db1

[test_host]
test151

[weblogic_hosts:children]
java1_host
java2_host

[database_hosts:children]
db1_host

[all:children]
dmz_host
proxy_host
weblogic_hosts
database_hosts
test_host

Next, we need to create an SSH user (this user will be used by Ansible to connect to remote hosts and perform defined tasks):

# create a new user, for example a user named ansible that belongs to a group named admin
$ sudo su - root
$ adduser ansible
$ groupadd admin
$ usermod -a -G admin ansible

# edit /etc/sudoers and add the admin group
## Allow root to run any commands anywhere
root    ALL=(ALL)           ALL
%admin  ALL=(ALL) NOPASSWD: ALL
# when using pipelining in Ansible disable requiretty
##Defaults    requiretty

# set-up ssh trust between Ansible clients
[ansible@hostname ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /home/ansible/.ssh/id_rsa.
Your public key has been saved in /home/ansible/.ssh/id_rsa.pub.

[ansible@different_hostname ~]$ mkdir -m 0700 .ssh
[ansible@different_hostname ~]$ install -m 0600 /dev/null .ssh/authorized_keys
[ansible@different_hostname ~]$ cat >> .ssh/authorized_keys
ssh-rsa ...
^C

# test ssh between Ansible clients
[ansible@hostname ~]$ ssh ansible@different_hostname hostname

# test Ansible
[ansible@hostname ansible]$ ansible all -m ping --inventory=inventories/development/demo_hosts.cfg
[ansible@hostname ansible]$ ansible all -a "/bin/echo hello" --inventory=inventories/development/demo_hosts.cfg
[ansible@hostname ansible]$ ansible all -m setup -a 'filter=*fqdn*' --inventory=inventories/development/demo_hosts.cfg

Linux Tools

Install some handy Linux Tools (such as Chrony, and Python Package Installer)

Used Variables:

demo_domain_name: "ms.com"
demo_domain_ips: "172.20.40.0/24"

dmz_hostname: "dmz"
dmz_ip_addr: "172.20.40.142"
dmz_fqdn: "{{ dmz_hostname }}.{{ demo_domain_name }}"

proxy_hostname: "proxy"
proxy_ip_addr: "172.20.40.143"
proxy_fqdn: "{{ proxy_hostname }}.{{ demo_domain_name }}"

java1_hostname: "java1"
java1_ip_addr: "172.20.40.144"
java1_fqdn: "{{ java1_hostname }}.{{ demo_domain_name }}"

java2_hostname: "java2"
java2_ip_addr: "172.20.40.146"
java2_fqdn: "{{ java2_hostname }}.{{ demo_domain_name }}"

db1_hostname: "db1"
db1_ip_addr: "172.20.40.145"
db1_fqdn: "{{ db1_hostname }}.{{ demo_domain_name }}"

test_hostname: "test151"
test_ip_addr: "172.20.40.151"
test_fqdn: "{{ test_hostname }}.{{ demo_domain_name }}"

linux_tool_packages:
  - chrony
  - dos2unix
  - mlocate
  - openldap-clients
  - python-pip
  - python-wheel
  - acme-tiny
  - traceroute
  - yum-utils  

python_libraries:
  - pyopenssl
  - pymysql
  - psycopg2-binary
  - passlib

hardening_packages:
  - firewalld
  - policycoreutils
  - selinux-policy
  - selinux-policy-targeted
  - libselinux-utils
  - libselinux-python
  - setroubleshoot-server
  - setools-console
  - mcstrans
  - fail2ban-firewalld

# chrony server pool
chrony_server_0: "server 0.nl.pool.ntp.org iburst"
chrony_server_1: "server 1.nl.pool.ntp.org iburst"
chrony_server_2: "server 2.nl.pool.ntp.org iburst"
chrony_server_3: "server 3.nl.pool.ntp.org iburst"

# chrony server
chrony_server: "server {{ dmz_fqdn }} iburst"

# chrony, allow NTP client access from local network
chrony_allow: "172.20/16"

Configure NTP

Install Chrony

On the DMZ node, we configure chrony (/etc/chrony.conf) by using the following template:

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
{{ chrony_server_0 }}
{{ chrony_server_1 }}
{{ chrony_server_2 }}
{{ chrony_server_3 }}

# UDP port on which the server understands NTP requests
port 123

# Ignore stratum in source selection.
stratumweight 0

# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift

# Enable kernel RTC synchronization.
rtcsync

# In first three updates step the system clock instead of slew
# if the adjustment is larger than 10 seconds.
makestep 10 3

# Allow NTP client access from local network.
allow {{ chrony_allow }}

# Listen for commands only on localhost.
bindcmdaddress 127.0.0.1
bindcmdaddress ::1

# Serve time even if not synchronized to any NTP server.
#local stratum 10

keyfile /etc/chrony.keys

# Specify the key used as password for chronyc.
commandkey 1

# Generate command key if missing.
generatecommandkey

# Disable logging of client accesses.
noclientlog

# Send a message to syslog if a clock adjustment is larger than 0.5 seconds.
logchange 0.5

logdir /var/log/chrony
#log measurements statistics tracking

details about public NTP servers can be found here (the pool.ntp.org project is a big virtual cluster of timeservers providing a reliable NTP service).

On the other nodes we use the following template:

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
{{ chrony_server }}

# UDP port on which the server understands NTP requests
port 123
 
# Ignore stratum in source selection.
stratumweight 0
 
# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift
 
# Enable kernel RTC synchronization.
rtcsync
 
# In first three updates step the system clock instead of slew
# if the adjustment is larger than 10 seconds.
makestep 10 3
 
# Allow NTP client access from local network.
#allow 192.168/16
 
# Listen for commands only on localhost.
bindcmdaddress 127.0.0.1
bindcmdaddress ::1
 
# Serve time even if not synchronized to any NTP server.
#local stratum 10
 
keyfile /etc/chrony.keys
 
# Specify the key used as password for chronyc.
commandkey 1
 
# Generate command key if missing.
generatecommandkey
 
# Disable logging of client accesses.
noclientlog
 
# Send a message to syslog if a clock adjustment is larger than 0.5 seconds.
logchange 0.5
 
logdir /var/log/chrony
#log measurements statistics tracking

Start chrony.

To combine the above in an Ansible playbook, we have the following

- name: install extra packages for enterprise linux repository configuration
  yum:
    name=https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    state=present

- name: install tools
  yum:
    name: "{{ linux_tool_packages }}"
    state: present
    
- name: upgrade pip
  pip:
    name: pip
    state: latest
  
- name: install python libraries
  pip:
    name: "{{ python_libraries }}"
    state: latest
  
- name: edit server pool /etc/chrony.conf
  template: 
    src: roles/demo_linux_tools/templates/chrony_server_pool_conf.j2
    dest: /etc/chrony.conf
    owner: root
    group: root
    mode: 0644
  register: chrony_conf_dmz_changed
  when: inventory_hostname == dmz_hostname
  
- name: edit server /etc/chrony.conf
  template: 
    src: roles/demo_linux_tools/templates/chrony_server_conf.j2
    dest: /etc/chrony.conf
    owner: root
    group: root
    mode: 0644
  register: chrony_conf_proxy_changed
  when: inventory_hostname != dmz_hostname

- name: enable and start chrony service
  systemd:
    name: chronyd
    state: started
    enabled: yes
  register: chrony_started
  
- name: restart chrony service
  systemd:
    name: chronyd
    state: restarted
  when: chrony_started is success and (chrony_conf_dmz_changed is changed or chrony_conf_proxy_changed is changed)

The python_libraries variable contains a list of python packages that are needed by Ansible modules, such as postgresql_db (see the requirements section of a particular module for more information).

DMZ

To set-up the DMZ, we use the following:

Used variables:

# ssh
sshd_permit_root_login: "yes"
sshd_allow_groups: "root admin {{ ds_operator_group }}"
sshd_authorized_keys_command: "/usr/local/bin/get-sshkey-from-ldap.sh"

# selinux configuratie
selinux_state: "permissive"
#selinux_state: "enforcing"
selinux_policy: "targeted"

Configure the sshd, here we set variables such as sshd_permit_root_login, sshd_allow_groups, and sshd_authorized_keys_command. To edit these parameters we create a template based on the existing sshd_config file, for example,

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/bin:/usr/bin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

# If you want to change the port on a SELinux system, you have to tell
# SELinux about this change.
# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
#
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#AllowGroups {{ sshd_allow_groups }}
PermitRootLogin {{ sshd_permit_root_login }}
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile .ssh/authorized_keys

#AuthorizedPrincipalsFile none

AuthorizedKeysCommand {{ sshd_authorized_keys_command }}
AuthorizedKeysCommandUser {{ root_user }}
...

The authorized keys command calls the following script

#!/bin/sh

ldapsearch -h {{ db1_fqdn }} -p {{ ds_directory_server_network_port }}  -b "{{ ds_suffix }}" -x "(&(objectClass=ldapPublicKey)(uid=$1))" "sshPublicKey" | sed -n '/^ /{H;d};/sshPublicKey:/x;$g;s/\n *//g;s/sshPublicKey: //gp'

which obtains the ssh public key from the Directory Server for a given user.

Add an extra sudoers in the /etc/sudoers.d directory

## Command Aliases
## These are groups of related commands

## Networking
Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptables, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

## Installation and management of software
Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum

## Services
Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig, /usr/bin/systemctl start, /usr/bin/systemctl stop, /usr/bin/systemctl reload, /usr/bin/systemctl restart, /usr/bin/systemctl status, /usr/bin/systemctl enable, /usr/bin/systemctl disable

## Updating the locate database
Cmnd_Alias LOCATE = /usr/bin/updatedb

## Storage
Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
Cmnd_Alias DRIVERS = /sbin/modprobe

## Allows members of the 'operators' group to run networking, software, service management apps and more.
%operators ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

Edit Kernel parameters, for example,

# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
# net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536

# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

# semaphores
kernel.sem = 250 32000 100 128
# shared memory
kernel.shmmni = 4096

# file handles
fs.file-max = 6815744

# asynchronous i/o
fs.aio-max-nr = 1048576

# disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

# default range of IP port numbers that are allowed for TCP and UDP traffic
net.ipv4.ip_local_port_range = 9000 65535

# increase TCP max buffer size (depending on the type of NIC and the round-trip time these values can be changed)
# Maximum TCP Receive Window
net.core.rmem_max = 8388608
net.core.rmem_default = 8388608
# Maximum TCP Send Window
net.core.wmem_max = 8388608
net.core.wmem_default = 8388608
#  memory reserved for TCP receive buffers (vector of 3 integers: [min, default, max])
net.ipv4.tcp_rmem = 4096 87380 8388608
# memory reserved for TCP send buffers (vector of 3 integers: [min, default, max])
net.ipv4.tcp_wmem = 4096 87380 8388608

# increase the length of the processor input queue
net.core.netdev_max_backlog = 30000
# maximum amount of memory buffers (could be set equal to net.core.rmem_max and net.core.wmem_max)
net.core.optmem_max = 20480
# socket of the listen backlog
net.core.somaxconn = 1024

# tcp selective acknowledgements (disable them on high-speed networks)
net.ipv4.tcp_sack = 1
net.ipv4.tcp_dsack = 1
# Timestamps add 12 bytes to the TCP header
net.ipv4.tcp_timestamps = 1
# Support for large TCP Windows - Needs to be set to 1 if the Max TCP Window is over 65535
net.ipv4.tcp_window_scaling = 1
# The interval between the last data packet sent (simple ACKs are not considered data) and the first keepalive probe
net.ipv4.tcp_keepalive_time = 300
# The interval between subsequential keepalive probes, regardless of what the connection has exchanged in the meantime
net.ipv4.tcp_keepalive_intvl = 30
# The number of unacknowledged probes to send before considering the connection dead and notifying the application layer
net.ipv4.tcp_keepalive_probes = 5
# The time that must elapse before TCP/IP can release a closed connection and reuse its resources.
net.ipv4.tcp_fin_timeout = 30
# Size of the backlog connections queue.
net.ipv4.tcp_max_syn_backlog = 4096
# The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers.
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

# The percentage of how aggressively memory pages are swapped to disk
vm.swappiness = 0
# The percentage of main memory the pdflush daemon should write data out to the disk.
vm.dirty_background_ratio = 25
# The percentage of main memory the actual disk writes will take place.
vm.dirty_ratio = 20

Edit Linux resources, for example,

# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
#Also note that configuration files in /etc/security/limits.d directory,
#which are read in alphabetical order, override the settings in this
#file in case the domain is the same or more specific.
#That means for example that setting a limit for wildcard domain here
#can be overriden with a wildcard setting in a config file in the
#subdirectory, but a user specific setting here can be overriden only
#with a user specific setting in the subdirectory.
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

*               soft     nofile          8192
*               hard     nofile          8192

# End of file

Configure the Firewall. To this end, we use the firewalld module, and add services such as https, ntp, ldap.
Configure SELinux and set the httpd_can_network_connect and httpd_can_network_relay flags. To this end, we use the selinux and the seboolean modules.

To combine the above in an Ansible playbook, we have the following

- name: install hardening packages
  yum:
    name: "{{ hardening_packages }}"
    state: present

# secure ssh
- name: edit /etc/ssh/sshd_config
  template: 
    src: roles/demo_linux_hardening/templates/sshd_config.j2
    dest: /etc/ssh/sshd_config
    owner: root
    group: root
    mode: 0600
  register: sshd_edited

- name: add /usr/local/bin/get-sshkey-from-ldap.sh
  template:
    src: roles/demo_linux_hardening/templates/get_sshkey_from_ldap_sh.j2
    dest: "{{ sshd_authorized_keys_command }}"
    owner: root
    group: root
    mode: 0700

- name: restart sshd service
  systemd:
    name: sshd
    state: restarted
  when: sshd_edited is changed

# sudoers
- name: add /etc/sudoers.d/demo_sudoers
  template:
    src: roles/demo_linux_hardening/templates/demo_sudoers.j2
    dest: /etc/sudoers.d/demo_sudoers
    owner: root
    group: root
    mode: 0440

# kernel parameters
- name: set sysctl /etc/sysctl.d/10-default_sysctl.conf
  template: 
    src: roles/demo_linux_hardening/templates/default_sysctl_conf.j2
    dest: /etc/sysctl.d/10-default_sysctl.conf
    owner: root
    group: root
    mode: 0644

- name: enable ip forwarding
  sysctl:
    name: net.ipv4.ip_forward
    value: 1
    sysctl_file: /etc/sysctl.d/11-dmz_proxy_sysctl.conf
    sysctl_set: yes
    state: present
    reload: yes
  when: inventory_hostname == dmz_hostname or inventory_hostname == proxy_hostname

# linux resource limits
- name: set /etc/security/limits.conf
  template:
    src: roles/demo_linux_hardening/templates/limits_conf.j2
    dest: /etc/security/limits.conf
    owner: root
    group: root
    mode: 0644

# configure firewall
- name: check if iptables exists
  stat: path=/usr/lib/systemd/system/iptables.service
  register: iptables_status
    
- name: disable iptables
  service:
    name=iptables
    state=stopped
    enabled=no
  when: iptables_status.stat.exists
  
- name: check if firewalld exists
  stat: path=/usr/lib/systemd/system/firewalld.service
  register: firewalld_status

- name: enable firewalld
  systemd:
    name: firewalld
    state: started
    enabled: yes
  when: firewalld_status.stat.exists

- name: activate public zone by adding interface to public zone
  firewalld:
    zone: public
    interface: "{{ ansible_default_ipv4.interface }}"
    state: enabled
    permanent: yes
    immediate: yes
    
- name: remove dhcpv6-client service from zone public
  firewalld:
    zone: public
    service: dhcpv6-client
    state: disabled
    permanent: yes
    immediate: yes
    
- name: add http service to zone public
  firewalld:
    zone: public
    service: http
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == dmz_hostname

- name: add https service to zone public
  firewalld:
    zone: public
    service: https
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == dmz_hostname

- name: add ntp service to zone public
  firewalld:
    zone: public
    service: ntp
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == dmz_hostname
  
- name: add rich rule to forward all ipv4 traffic on port 80 to port 8080
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 forward-port port=80 protocol=tcp to-port={{ nginx_http_listen_port }}'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == dmz_hostname
  
- name: add rich rule to forward all ipv4 traffic on port 443 to port 8443
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 forward-port port=443 protocol=tcp to-port={{ nginx_listen_port }}'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == dmz_hostname

- name: add rich rule to accept request from certain host on port 8080
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port={{ apache_listen_port }} protocol=tcp source address={{ dmz_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == proxy_hostname
  
- name: add rich rule to accept request from java1 host on port 5556
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port=5556 protocol=tcp source address={{ java1_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == java1_hostname or inventory_hostname == java2_hostname
  
- name: add rich rule to accept request from java2 host on port 5556
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port=5556 protocol=tcp source address={{ java2_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == java1_hostname or inventory_hostname == java2_hostname
  
- name: add rich rule to accept request from java1 host on port 7001
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port=7001 protocol=tcp source address={{ java1_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == java1_hostname or inventory_hostname == java2_hostname
  
- name: add rich rule to accept request from java2 host on port 7001
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port=7001 protocol=tcp source address={{ java2_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == java1_hostname or inventory_hostname == java2_hostname
  
- name: add rich rule to accept request from proxy host on port 7001
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port=7001 protocol=tcp source address={{ proxy_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == java1_hostname or inventory_hostname == java2_hostname
  
- name: add rich rule to accept request from java1 host on port 9002-9003
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port=9002-9003 protocol=tcp source address={{ java1_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == java1_hostname or inventory_hostname == java2_hostname
  
- name: add rich rule to accept request from java2 host on port 9002-9003
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port=9002-9003 protocol=tcp source address={{ java2_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == java1_hostname or inventory_hostname == java2_hostname
  
- name: add rich rule to accept request from proxy host on port 9002-9003
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port=9002-9003 protocol=tcp source address={{ proxy_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == java1_hostname or inventory_hostname == java2_hostname

- name: add ldap service to zone public
  firewalld:
    zone: public
    service: ldap
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == db1_hostname
  
- name: add mysql port to zone public
  firewalld:
    zone: public
    port: 3306/tcp
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == db1_hostname
  
- name: add postgresql port to zone public
  firewalld:
    zone: public
    port: 5432/tcp
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == db1_hostname

- name: add rich rule to accept request from certain host on port 80
  firewalld:
    zone: public
    rich_rule: 'rule family=ipv4 port port={{ cmk_listen_port }} protocol=tcp source address={{ proxy_ip_addr }} accept'
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == test_hostname
  
- name: add ldap and ldaps to zone public
  firewalld:
    zone: public
    service: "{{ item }}"
    state: enabled
    permanent: yes
    immediate: yes
  when: inventory_hostname == test_hostname
  loop:
    - ldap
    - ldaps
    
# configure selinux
- name: enable selinux
  selinux:
    policy: "{{ selinux_policy }}"
    state: "{{ selinux_state }}"
  notify:
    - restart server
    - wait for server to restart

- name: set httpd_can_network_connect flag on and keep it persistent across reboots
  seboolean:
    name: httpd_can_network_connect
    state: yes
    persistent: yes
    
- name: set httpd_can_network_relay flag on and keep it persistent across reboots
  seboolean:
    name: httpd_can_network_relay
    state: yes
    persistent: yes


# handler that is called when seliux settings are changed
- name: restart server
  shell: sleep 2 && shutdown -r now "Ansible updates triggered"
  async: 1
  poll: 0
  ignore_errors: True
  
- name: wait for server to restart
  local_action: wait_for
    host={{ inventory_hostname }}
    port=22
    state=started
    delay=10
    timeout=300
  become: false

To check the firewall configuration we can use firewall-cmd (see here for more information):

[root@dmz ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: ssh https ntp http
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
        rule family="ipv4" forward-port port="443" protocol="tcp" to-port="8443"
        rule family="ipv4" forward-port port="80" protocol="tcp" to-port="8080"

More information on using firewalls and getting started with firewalld can be found here.

When configuring SELinux it is wise to first run SELinux in Permissive mode to obtain audit logging before setting it to Enforcing, i.e., access a certain host and check what is happening in the audit logging (/var/log/audit/audit.log), for example,

type=AVC msg=audit(1552124629.566:127): avc:  denied  { name_connect } for  pid=6566 comm="nginx" dest=8080 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket permissive=1
type=SYSCALL msg=audit(1552124629.566:127): arch=c000003e syscall=42 success=no exit=-115 a0=b a1=7f7c5ed40080 a2=10 a3=7fffb8e8f210 items=0 ppid=6565 pid=6566 auid=4294967295 uid=996 gid=993 euid=996 suid=996 fsuid=996 egid=993 sgid=993 fsgid=993 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552124629.566:127): proctitle=6E67696E783A20776F726B65722070726F63657373

Using: grep 1552124629.566:127 /var/log/audit/audit.log | audit2why gives:

type=AVC msg=audit(1552124629.566:127): avc:  denied  { name_connect } for  pid=6566 comm="nginx" dest=8080 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket permissive=1

Was caused by:
One of the following booleans was set incorrectly.
Description:
Allow httpd to can network connect

Allow access by executing:
# setsebool -P httpd_can_network_connect 1
Description:
Allow httpd to can network relay

Allow access by executing:
# setsebool -P httpd_can_network_relay 1

The output from audit2why indicates that we can allow Nginx to make proxy connections by enabling one or both of the httpd_can_network_relay and httpd_can_network_connect Boolean options. We can enable them either temporarily or permanently, the latter by adding the -P flag as shown in the output. The sesearch command provides more information about the Boolean options:

[root@dmz ~]# sesearch -A -s httpd_t -b httpd_can_network_connect
Found 1 semantic av rules:
   allow httpd_t port_type : tcp_socket name_connect ;

This output indicates that httpd_can_network_connect allows processes labeled with the httpd_t context to connect to all TCP socket types that have the port_type attribute.

[root@dmz ~]# seinfo -aport_type -x
   port_type
      afs3_callback_port_t
      afs_bos_port_t
      afs_fs_port_t
      afs_ka_port_t
      afs_pt_port_t
      afs_vl_port_t
      agentx_port_t
      amanda_port_t
      ...

[root@dmz ~]# sesearch -A -s httpd_t -b httpd_can_network_relay
Found 17 semantic av rules:
   allow httpd_t http_port_t : tcp_socket name_connect ;
   allow httpd_t squid_port_t : tcp_socket name_connect ;
   allow httpd_t ephemeral_port_type : tcp_socket name_connect ;
   allow httpd_t ftp_port_t : tcp_socket name_connect ;
   allow httpd_t gopher_client_packet_t : packet recv ;
   allow httpd_t gopher_client_packet_t : packet send ;
   allow httpd_t http_cache_port_t : tcp_socket name_connect ;
   allow httpd_t squid_client_packet_t : packet recv ;
   allow httpd_t squid_client_packet_t : packet send ;
   allow httpd_t http_cache_client_packet_t : packet recv ;
   allow httpd_t http_cache_client_packet_t : packet send ;
   allow httpd_t gopher_port_t : tcp_socket name_connect ;
   allow httpd_t http_client_packet_t : packet recv ;
   allow httpd_t http_client_packet_t : packet send ;
   allow httpd_t memcache_port_t : tcp_socket name_connect ;
   allow httpd_t ftp_client_packet_t : packet recv ;
   allow httpd_t ftp_client_packet_t : packet send ;

This output indicates that httpd_can_network_relay allows processes labeled with the httpd_t context to connect to ports of various types, including type http_port_t.

By default, the SELinux configuration does not allow to listen to TCP or UDP ports other than the default ones that are whitelisted in the http_port_t type:

[root@dmz ~]# semanage port -l | grep http_port_t
http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000
pegasus_http_port_t            tcp      5988

[root@dmz ~]# semanage port -a -t http_port_t -p tcp 8080
ValueError: Port tcp/8080 already defined

[root@dmz ~]# semanage port -l | grep 8080
http_cache_port_t              tcp      8080, 8118, 8123, 10001-10010

# semanage port -a -t http_port_t -p tcp 8001
# https://docs.ansible.com/ansible/latest/modules/seport_module.html
- name: allow a process to listen on tcp port 8001
  seport:
    ports: 8001
    proto: tcp
    setype: http_port_t
    state: present
    reload: yes

SELinux troubleshooting is further explained in the article: Basic SELinux Troubleshooting in CLI (using the commands ausearch -m AVC,USER_AVC, ausearch -m AVC,USER_AVC -ts recent, and journalctl -t setroubleshoot).

To configure Nginx we can use the following:

Used Variables

nginx_repo_name: "nginx-stable"
nginx_repo_baseurl: "http://nginx.org/packages/rhel/7/$basearch/"
nginx_repo_gpgkey: ""

nginx_http_listen_port: "8080"
nginx_listen_port: "8443"
nginx_server_name: "{{ dmz_fqdn }}"

nginx_ssl_session_timeout: "1d"
nginx_ssl_session_cache: "shared:SSL:50m"
nginx_ssl_session_tickets: "off"
nginx_ssl_protocols: "TLSv1.2"
nginx_ssl_ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
nginx_ssl_prefer_server_ciphers: "on"
nginx_hsts_header: "Strict-Transport-Security max-age=15768000"

nginx_dir: "/etc/nginx"
nginx_conf_dir: "{{nginx_dir }}/conf.d"
nginx_conf: "{{ nginx_dir }}/nginx.conf"
nginx_apache_web_server_conf: "{{ nginx_conf_dir }}/apache_web_server.conf"
nginx_systemd_service: "/etc/systemd/system/nginx.service"

# nginx ssl using self-signed certificates
openssl_ca_private_key_passphrase: "password"
openssl_cipher: "aes256"
openssl_digest: "sha256"
openssl_key_size: "2048"
openssl_days_valid: "365"
openssl_subject_country: "Some Country"
openssl_subject_state_or_province_name: "Some Province"
openssl_subject_locality_name: "Some Locality"
openssl_subject_organization_name: "Middleware Snippets"
openssl_subject_organizational_unit_name: "Blogging"
openssl_subject_common_name: "{{ ansible_fqdn }}"
openssl_subject: "/C={{ openssl_subject_country }}/ST={{ openssl_subject_state_or_province_name }}/L={{ openssl_subject_locality_name }}/O={{ openssl_subject_organization_name }}/OU={{ openssl_subject_organizational_unit_name }}/CN={{ openssl_subject_common_name }}"
openssl_subject_client: "/CN=client"

openssl_certificates_directory: "{{ nginx_dir }}/certificates"
openssl_ca_certificates_directory: "{{ openssl_certificates_directory }}/ca"
openssl_server_certificates_directory: "{{ openssl_certificates_directory }}/server"
openssl_client_certificates_directory: "{{ openssl_certificates_directory }}/client"

openssl_ca_private_key_path: "{{openssl_ca_certificates_directory }}/ca-key.pem"
openssl_ca_certificate_path: "{{openssl_ca_certificates_directory }}/ca.pem"
openssl_server_private_key_path: "{{ openssl_server_certificates_directory }}/server-key.pem"
openssl_server_certificate_request_path: "{{ openssl_server_certificates_directory }}/server.csr"
openssl_server_certificate_path: "{{ openssl_server_certificates_directory }}/server-cert.pem"
openssl_client_private_key_path: "{{ openssl_client_certificates_directory }}/key.pem"
openssl_client_certificate_request_path: "{{ openssl_client_certificates_directory }}/client.csr"
openssl_client_extended_key_usage_path: "{{ openssl_client_certificates_directory }}/extfile.cnf"
openssl_client_certificate_path: "{{ openssl_client_certificates_directory }}/cert.pem"

SSL configuration such as cipher suites can obtained from the Mozilla SSL Configuration Generator.

Add Nginx repository.
Install Nginx.

For testing purposes generate self-signed certificates

#!/bin/sh

echo 'REMOVING OLD CERTIFICATES'
rm -rf {{ openssl_certificates_directory }}


echo 'CREATING DIRECTORIES'
mkdir -p {{ openssl_ca_certificates_directory }}
mkdir -p {{ openssl_server_certificates_directory }}
mkdir -p {{ openssl_client_certificates_directory }}
 

echo 'GENERATING CA CERTIFICATES'
# https://www.openssl.org/docs/manmaster/man1/genrsa.html
openssl genrsa -{{ openssl_cipher }} -passout pass:{{ openssl_ca_private_key_passphrase }} -out {{ openssl_ca_private_key_path }} {{ openssl_key_size }}
# https://docs.ansible.com/ansible/latest/modules/openssl_privatekey_module.html#openssl-privatekey-module
#- openssl_privatekey:
#    path: "{{ openssl_ca_private_key_path }}"
#    passphrase: "{{ openssl_ca_private_key_passphrase }}"
#    cipher: "{{ openssl_cipher }}"
#    size: "{{ openssl_key_size }}"
#    mode: 0400

# https://www.openssl.org/docs/manmaster/man1/req.html
openssl req -new -x509 -days {{ openssl_days_valid }} -passin pass:{{ openssl_ca_private_key_passphrase }} -key {{ openssl_ca_private_key_path }} -{{ openssl_digest }} -out {{ openssl_ca_certificate_path }} -subj "{{ openssl_subject }}"
# https://docs.ansible.com/ansible/latest/modules/openssl_csr_module.html#openssl-csr-module
#- openssl_csr:
#    path: "{{ openssl_ca_certificate_path }}"
#    privatekey_path: "{{ openssl_ca_private_key_path }}"
#    privatekey_passphrase: "{{ openssl_ca_private_key_passphrase }}"
#    digest: "{{ openssl_digest }}"
#    country_name: "{{ openssl_subject_country }}"
#    state_or_province_name: "{{ openssl_subject_state_or_province_name }}"
#    locality_name: "{{ openssl_subject_locality_name }}"
#    organization_name: "{{ openssl_subject_organization_name }}"
#    organizational_unit_name: "{{ openssl_subject_organizational_unit_name }}"
#    common_name: "{{ openssl_subject_common_name }}"
#    mode: 0444
 

echo 'GENERATING SERVER CERTIFICATES'
# https://www.openssl.org/docs/manmaster/man1/genrsa.html
openssl genrsa -out {{ openssl_server_private_key_path }} {{ openssl_key_size }}
# https://docs.ansible.com/ansible/latest/modules/openssl_privatekey_module.html#openssl-privatekey-module
#- openssl_privatekey:
#    path: "{{ openssl_server_private_key_path }}"
#    cipher: "{{ openssl_cipher }}"
#    size: "{{ openssl_key_size }}"
#    mode: 0400

# https://www.openssl.org/docs/manmaster/man1/req.html
openssl req -new -key {{ openssl_server_private_key_path }} -{{ openssl_digest }} -out {{ openssl_server_certificate_request_path }} -subj "{{ openssl_subject }}"
# https://docs.ansible.com/ansible/latest/modules/openssl_csr_module.html#openssl-csr-module
#- openssl_csr:
#    path: "{{ openssl_server_certificate_request_path }}"
#    privatekey_path: "{{ openssl_server_private_key_path }}"
#    digest: "{{ openssl_digest }}"
#    country_name: "{{ openssl_subject_country }}"
#    state_or_province_name: "{{ openssl_subject_state_or_province_name }}"
#    locality_name: "{{ openssl_subject_locality_name }}"
#    organization_name: "{{ openssl_subject_organization_name }}"
#    organizational_unit_name: "{{ openssl_subject_organizational_unit_name }}"
#    common_name: "{{ openssl_subject_common_name }}"
#    mode: 0444

# https://www.openssl.org/docs/manmaster/man1/x509.html
openssl x509 -req -days {{ openssl_days_valid }} -{{ openssl_digest }} -passin pass:{{ openssl_ca_private_key_passphrase }} -in {{ openssl_server_certificate_request_path }} -CA {{ openssl_ca_certificate_path }} -CAkey {{ openssl_ca_private_key_path }} -CAcreateserial -out {{ openssl_server_certificate_path }}
# https://docs.ansible.com/ansible/latest/modules/openssl_certificate_module.html#openssl-certificate-module
#- openssl_certificate:
#    path: "{{ openssl_server_certificate_path }}"
#    csr_path: "{{ openssl_server_certificate_request_path }}"
#    ownca_path: "{{ openssl_ca_certificate_path }}"
#    ownca_privatekey_path: "{{ openssl_ca_private_key_path }}"
#    ownca_privatekey_passphrase: "{{ openssl_ca_private_key_passphrase }}"
#    provider: ownca
#    invalid_at: YYYYmmddHHMMSS
#    invalid_at: 20200310235959
#    mode: 0444

 
echo 'GENERATING CLIENT CERTIFICATES'
# https://www.openssl.org/docs/manmaster/man1/genrsa.html
openssl genrsa -out {{ openssl_client_private_key_path }} {{ openssl_key_size }}
# https://www.openssl.org/docs/manmaster/man1/req.html
openssl req -new -key {{ openssl_client_private_key_path }} -{{ openssl_digest }} -out {{ openssl_client_certificate_request_path }} -subj "{{ openssl_subject_client }}"
# https://www.openssl.org/docs/manmaster/man1/x509.html
echo extendedKeyUsage = clientAuth > {{ openssl_client_extended_key_usage_path }}
openssl x509 -req -days {{ openssl_days_valid }} -{{ openssl_digest }} -passin pass:{{ openssl_ca_private_key_passphrase }} -in {{ openssl_client_certificate_request_path }} -CA {{ openssl_ca_certificate_path }} -CAkey {{ openssl_ca_private_key_path }} -CAcreateserial -out {{ openssl_client_certificate_path }} -extfile {{ openssl_client_extended_key_usage_path }}
 

echo 'CHANGING PERMISSIONS'
chmod 0400 {{ openssl_ca_private_key_path }} {{ openssl_server_private_key_path }} {{ openssl_client_private_key_path }}
chmod 0444 {{ openssl_ca_certificate_path }} {{ openssl_server_certificate_path }} {{ openssl_client_certificate_path }}
 

echo 'REMOVING CERTIFICATE SIGNING REQUESTS'
rm -f {{ openssl_server_certificate_request_path }} {{ openssl_client_certificate_request_path }}

Edit the main Nginx configuration file:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include {{ nginx_apache_web_server_conf }};
}

Add the server configuration

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream apache {
    # Health-monitored upstream groups must have a zone defined
    zone apache 64k;

    # Use IP Hash for session persistence
    ip_hash;
    
    # List of Apache Web Servers
    server {{ proxy_fqdn }}:{{ apache_listen_port }} weight=100 max_fails=5 fail_timeout=300;
}

server {
    listen {{ nginx_http_listen_port }};
    server_name {{ nginx_server_name }};
    return 301 https://{{ nginx_server_name }}$request_uri;
}

server {
    listen {{ nginx_listen_port }} ssl http2;
    server_name {{ nginx_server_name }};
    
    ssl_certificate {{ openssl_server_certificate_path }};
    ssl_certificate_key {{ openssl_server_private_key_path }};
    ssl_trusted_certificate {{ openssl_ca_certificate_path }};
    ssl_session_timeout {{ nginx_ssl_session_timeout }};
    ssl_session_cache {{ nginx_ssl_session_cache }};
    ssl_session_tickets {{ nginx_ssl_session_tickets }};

    ssl_protocols {{ nginx_ssl_protocols }};
    ssl_ciphers {{ nginx_ssl_ciphers }};
    ssl_prefer_server_ciphers {{ nginx_ssl_prefer_server_ciphers }};

    # HSTS (ngx_http_headers_module is required)
    add_header {{ nginx_hsts_header }};
   
    location / {
        proxy_pass                            http://apache;
        proxy_buffering                       off;
        proxy_set_header Host                 $http_host;
        proxy_set_header X-Real-IP            $remote_addr;
        proxy_set_header X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto    $scheme;
        proxy_set_header WL-Proxy-SSL         true;
    }
}

To start NGINX using systemd, we can use

[Unit]
Description=Nginx Web Server
After=network.target

[Service]
User=root
Group=root
Type=forking
ExecStart=/usr/sbin/nginx
ExecStop=/usr/sbin/nginx -s stop
ExecReload=/usr/sbin/nginx -s reload
PIDFile=/var/run/nginx.pid
Restart=on-failure
RestartSec=1

[Install]
WantedBy=default.target

To combine the above in an Ansible playbook, we have the following:

- name: add nginx stable repository
  yum_repository:
    name: "{{ nginx_repo_name }}"
    description: Nginx Stable Repo
    baseurl: "{{ nginx_repo_baseurl }}"
    gpgcheck: no
    enabled: yes

- name: install nginx
  yum:
    name: nginx
    state: present
    enablerepo: "{{ nginx_repo_name }}"
    
#- name: check if server certificate is valid
#  openssl_certificate:
#    path: "{{ openssl_server_certificate_path }}"
#    provider: assertonly
#    has_expired: false
#  ignore_errors: true
#  register: validity_check
  
- name: check if server certificate is valid
  command: /bin/sh -c "openssl x509 -checkend $(( 86400 * 30 )) -in {{ openssl_server_certificate_path }}"
  register: validity_check
  changed_when: false
  failed_when: "'Certificate will expire' in validity_check.stdout or 'unable to load certificate' in validity_check.stderr"
  ignore_errors: true
  
- name: debug check if server certificate is valid
  debug: 
    var=validity_check
  changed_when: false
    
- name: edit /etc/nginx/generate_certificates.sh
  template: 
    src: roles/demo_nginx/templates/generate_certificates_sh.j2
    dest: "{{ nginx_dir }}/generate_certificates.sh"
    owner: root
    group: root
    mode: 0744
  register: generate_certificates_sh_created
  when: validity_check is failed
  
- name: run custom task to get a new, valid certificate in case the validity check failed
  command: /bin/sh -c "{{ nginx_dir }}/generate_certificates.sh"
  register: nginx_certs_changed
  when: validity_check is failed and generate_certificates_sh_created is success
  
- name: remove /etc/nginx/generate_certificates.sh
  file:
    path: "{{ nginx_dir }}/generate_certificates.sh"
    state: absent
  when: validity_check is failed
    
- name: edit /etc/nginx/nginx.conf
  template: 
    src: roles/demo_nginx/templates/nginx_conf.j2
    dest: "{{ nginx_conf }}"
    owner: root
    group: root
    mode: 0644
  register: nginx_conf_changed

- name: edit /etc/nginx/conf.d/apache_web_server.conf
  template: 
    src: roles/demo_nginx/templates/apache_web_server_conf.j2
    dest: "{{ nginx_apache_web_server_conf }}"
    owner: root
    group: root
    mode: 0644
  register: apache_web_server_conf_changed
  
- name: edit /etc/systemd/system/nginx.service
  template: 
    src: roles/demo_nginx/templates/nginx_service.j2
    dest: "{{ nginx_systemd_service }}"
    owner: root
    group: root
    mode: 0644
  register: nginx_service_changed
  
- name: start nginx service
  systemd:
    name: nginx
    state: started
    enabled: yes
  register: nginx_started
  
- name: restart nginx
  systemd:
    name: nginx
    state: restarted
  when: nginx_started is not changed and (nginx_conf_changed is changed or apache_web_server_conf_changed is changed or nginx_certs_changed is changed)

Load Balancer

To set-up the load balancer, we use the following:

Used variables

apache_dir: "/etc/httpd"
apache_httpd_conf_dir: "{{ apache_dir }}/conf"
apache_extra_conf_dir: "{{ apache_dir }}/conf.d"
apache_modules_conf_dir: "{{ apache_dir }}/conf.modules.d"

apache_httpd_conf_path: "{{ apache_httpd_conf_dir }}/httpd.conf"
apache_mpm_module_conf_path: "{{ apache_modules_conf_dir }}/00-mpm.conf"
apache_proxy_module_conf_path: "{{ apache_modules_conf_dir }}/00-proxy.conf"

apache_listen_port: "8080"
apache_server_name: "{{ proxy_fqdn }}"

# ServerLimit = MaxRequestWorkers / ThreadsPerChild
# MaxSpareThreads = MaxRequestWorkers
# ThreadLimit = ThreadsPerChild
# MaxConnectionsPerChild = 0
apache_mpm_server_limit: "40"
apache_mpm_thread_limit: "25"
apache_mpm_start_servers: "2"
apache_mpm_max_request_workers: "1000"
apache_mpm_max_spare_threads: "1000"
apache_mpm_min_spare_threads: "25"
apache_mpm_threads_per_child: "25"
apache_mpm_max_connections_per_child: "0"

Install Apache Httpd.
Edit httpd.conf by setting the Listen and the ServerName directives.

Configure a multi-processing module, for example,

# Select the MPM module which should be used by uncommenting exactly
# one of the following LoadModule lines:

# prefork MPM: Implements a non-threaded, pre-forking web server
# See: http://httpd.apache.org/docs/2.4/mod/prefork.html
#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

# worker MPM: Multi-Processing Module implementing a hybrid
# multi-threaded multi-process web server
# See: http://httpd.apache.org/docs/2.4/mod/worker.html
#
#LoadModule mpm_worker_module modules/mod_mpm_worker.so

# event MPM: A variant of the worker MPM with the goal of consuming
# threads only for connections with active processing
# See: http://httpd.apache.org/docs/2.4/mod/event.html
#
LoadModule mpm_event_module modules/mod_mpm_event.so

#
# Server-Pool Management (MPM specific)
# 


# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxRequestWorkers: maximum number of server processes allowed to start
# MaxConnectionsPerChild: maximum number of connections a server process serves
#                         before terminating
<IfModule mpm_prefork_module>
    StartServers             5
    MinSpareServers          5
    MaxSpareServers         10
    MaxRequestWorkers      250
    MaxConnectionsPerChild   0
</IfModule>


# worker MPM
# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestWorkers: maximum number of worker threads
# MaxConnectionsPerChild: maximum number of connections a server process serves before terminating

# With the following configuration, 1000 clients can connect simultaneously.
# When this is not sufficient, we must increase the MaxRequestWorkers directive. 
# When doing this, we also need to adjust the following directives 
# ServerLimit = MaxRequestWorkers / ThreadsPerChild
# MaxSpareThreads = MaxRequestWorkers
# ThreadLimit = ThreadsPerChild
# MaxConnectionsPerChild = 0
<IfModule mpm_worker_module>
    ServerLimit            {{ apache_mpm_server_limit }}
    ThreadLimit            {{ apache_mpm_thread_limit }}
    StartServers           {{ apache_mpm_start_servers }}
    MaxRequestWorkers      {{ apache_mpm_max_request_workers }}
    MaxSpareThreads        {{ apache_mpm_max_spare_threads }}
    MinSpareThreads        {{ apache_mpm_min_spare_threads }}
    ThreadsPerChild        {{ apache_mpm_threads_per_child }}
    MaxConnectionsPerChild {{ apache_mpm_max_connections_per_child }}
</IfModule>


# event MPM
# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestWorkers: maximum number of worker threads
# MaxConnectionsPerChild: maximum number of connections a server process serves before terminating

# With the following configuration, 1000 clients can connect simultaneously.
# When this is not sufficient, we must increase the MaxRequestWorkers directive. 
# When doing this, we also need to adjust the following directives 
# ServerLimit = MaxRequestWorkers / ThreadsPerChild
# MaxSpareThreads = MaxRequestWorkers
# ThreadLimit = ThreadsPerChild
# MaxConnectionsPerChild = 0
<IfModule mpm_event_module>
    ServerLimit            {{ apache_mpm_server_limit }}
    ThreadLimit            {{ apache_mpm_thread_limit }}
    StartServers           {{ apache_mpm_start_servers }}
    MaxRequestWorkers      {{ apache_mpm_max_request_workers }}
    MaxSpareThreads        {{ apache_mpm_max_spare_threads }}
    MinSpareThreads        {{ apache_mpm_min_spare_threads }}
    ThreadsPerChild        {{ apache_mpm_threads_per_child }}
    MaxConnectionsPerChild {{ apache_mpm_max_connections_per_child }}
</IfModule>


# The maximum number of free Kbytes that every allocator is allowed
# to hold without calling free(). In threaded MPMs, every thread has its own
# allocator. When not set, or when set to zero, the threshold will be set to
# unlimited.
<IfModule !mpm_netware_module>
    MaxMemFree            2048
</IfModule>
<IfModule mpm_netware_module>
    MaxMemFree             100
</IfModule>

Configure the mod_proxy module, for example,

LoadModule proxy_module modules/mod_proxy.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
LoadModule lbmethod_heartbeat_module modules/mod_lbmethod_heartbeat.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_express_module modules/mod_proxy_express.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule proxy_fdpass_module modules/mod_proxy_fdpass.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so

<VirtualHost *:{{ apache_listen_port }}>

    ProxyRequests Off
    ProxyPreserveHost On
    
    ProxyPass /console http://java1.ms.com:7001/console
    ProxyPassReverse /console http://java1.ms.com:7001/console
    
    ProxyPass /consolehelp http://java1.ms.com:7001/consolehelp
    ProxyPassReverse /consolehelp http://java1.ms.com:7001/consolehelp
 
    ProxyPass /javaee7 balancer://javaee7
    <Proxy balancer://javaee7>
        BalancerMember http://java1.ms.com:9002/javaee7
        BalancerMember http://java2.ms.com:9003/javaee7
    </Proxy>    
 
    ProxyPass /SessionTest balancer://sessiontest
    ProxyPassReverse /SessionTest balancer://sessiontest 
    Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
    <Proxy balancer://sessiontest>
        BalancerMember http://java1.ms.com:9002/SessionTest route=1
        BalancerMember http://java2.ms.com:9003/SessionTest route=2
        ProxySet stickysession=ROUTEID
    </Proxy>
    
    ProxyPass /support http://test151.ms.com:80/support
    ProxyPassReverse /support http://test151.ms.com:80/support
 
    <Location /balancer-manager>
        SetHandler balancer-manager
        Order Deny,Allow
        Allow from all
    </Location>

</VirtualHost>

To combine the above in an Ansible playbook, we have the following:

- name: install apache_httpd 
  yum:
    name: httpd
    state: present

- name: edit /etc/httpd/conf/httpd.conf
  template: 
    src: roles/demo_apache_httpd/templates/httpd_conf.j2
    dest: "{{ apache_httpd_conf_path }}"
    owner: root
    group: root
    mode: 0644
  register: httpd_conf_changed
  
- name: edit /etc/httpd/conf.modules.d/00-mpm.conf
  template: 
    src: roles/demo_apache_httpd/templates/httpd_00_mpm_conf.j2
    dest: "{{ apache_mpm_module_conf_path }}"
    owner: root
    group: root
    mode: 0644
  register: httpd_mpm_conf_changed
  
- name: edit /etc/httpd/conf.modules.d/00-proxy.conf
  template: 
    src: roles/demo_apache_httpd/templates/httpd_00_proxy_conf.j2
    dest: "{{ apache_proxy_module_conf_path }}"
    owner: root
    group: root
    mode: 0644
  register: httpd_proxy_conf_changed
    
- name: start httpd service
  systemd:
    name: httpd
    state: started
    enabled: yes
  register: httpd_started
  
- name: restart httpd
  systemd:
    name: httpd
    state: restarted
  when: httpd_started is not changed and (httpd_conf_changed is changed or httpd_mpm_conf_changed is changed or httpd_proxy_conf_changed is changed)

JavaEE Application Server

To set-up WebLogic using Ansible can be found in various posts, such as WebLogic Scripting for Ansible or Working with the ConfigurationManager and DeploymentManager.

Data Server

MySQL/MariaDB

To set-up the MySQL/MariaDB database we can use the following steps:

Used variables:

mariadb_repo_name: "MariaDB"
mariadb_repo_baseurl: "http://yum.mariadb.org/10.3/centos7-amd64"
mariadb_repo_gpgkey: "https://yum.mariadb.org/RPM-GPG-KEY-MariaDB"

mariadb_packages:
  - MariaDB-server
  - MariaDB-client

mariadb_conf_dir: "/etc/my.cnf.d/"
mariadb_server_conf_path: "/etc/my.cnf.d/server.cnf"
mariadb_server_mycnf_path: "/etc/my.cnf"
mariadb_client_mycnf_path: "/root/.my.cnf"

mariadb_anonymous_user: ""
mariadb_root_password: "{{ overall_password }}"
mariadb_root_privileges: "*.*:ALL,GRANT"

# example database
mariadb_example_database: "exampledb"
mariadb_example_user: "example"
mariadb_example_password: "set_in_inventory"
mariadb_example_privileges: "{{ mariadb_example_database }}.*:ALL,GRANT"
mariadb_example_url: "mariadb://{{ mariadb_example_user }}:{{ mariadb_example_password }}@{{ db1_fqdn }}/{{ mariadb_example_database }}"

Add the MariaDB repository.
Install MariaDB (or MySQL).

Configure the database service by using the following templates

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
[mysqld]
!includedir {{ mariadb_conf_dir }}

#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#

# this is read by the standalone daemon and embedded servers
[server]

# this is only for the mysqld standalone daemon
[mysqld]
bind-address = {{ db1_fqdn }}
default-storage-engine = innodb
innodb_file_per_table
collation-server = utf8_general_ci
init-connect = 'SET NAMES utf8'
character-set-server = utf8
# https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_autocommit
autocommit = 0

#
# * Galera-related settings
#
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0

# this is only for embedded server
[embedded]

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

# This group is only read by MariaDB-10.3 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand
[mariadb-10.3]

Start the database service.
Secure the database service. On the command-line, we would use mysql_secure_installation, Ansible provides the mysql_db and mysql_user modules. Note that MySQL server installs with a default login_user of root and no password. To secure this user as part of an idempotent playbook, we must create two tasks: the first must change the root user's password, without providing any login_user/login_password details. The second must drop a ~/.my.cnf file containing the new root credentials. Subsequent runs of the playbook will then succeed by reading the new credentials from the file. The template (my_cnf.j2) for the ~/.my.cnf file has the following contents
```
[client]
user=root
password={{ mariadb_root_password }}
```
Create an application database and user

To combine the above in an Ansible playbook, we have the following:

- name: add mariadb stable repository
  yum_repository:
    name: "{{ mariadb_repo_name }}"
    description: MariaDB Stable Repo
    baseurl: "{{ mariadb_repo_baseurl }}"
    gpgcheck: yes
    enabled: yes
    gpgkey: "{{ mariadb_repo_gpgkey }}"

- name: install mariadb
  yum:
    name: "{{ mariadb_packages }}"
    state: present
    enablerepo: "{{ mariadb_repo_name }}"

- name: create /etc/my.cnf
  template:
    src: roles/demo_mariadb/templates/server_my_cnf.j2
    dest: "{{ mariadb_server_mycnf_path }}"
    owner: root
    group: root
    mode: 0644
  register: mariadb_server_mycnf_changed

- name: create /etc/my.cnf.d/server.cnf
  template:
    src: roles/demo_mariadb/templates/server_cnf.j2
    dest: "{{ mariadb_server_conf_path }}"
    owner: root
    group: root
    mode: 0644
  register: mariadb_server_conf_changed

- name: enable and start mariadb service
  systemd:
    name: mariadb
    state: started
    enabled: yes
  register: mariadb_started

- name: delete anonymous user
  mysql_user:
    user: "{{ mariadb_anonymous_user }}"
    host: "{{ db1_fqdn }}"
    login_host: "{{ db1_fqdn }}"
    state: absent

- name: delete anonymous user for localhost
  mysql_user:
    user: "{{ mariadb_anonymous_user }}"
    login_host: "{{ db1_fqdn }}"
    state: absent

- name: remove the test database
  mysql_db:
    db: test
    login_host: "{{ db1_fqdn }}"
    state: absent

- name: change root password
  mysql_user:
    name: root
    password: "{{ mariadb_root_password }}"
    priv: "{{ mariadb_root_privileges }}"
    host: "{{ db1_fqdn }}"
    login_host: "{{ db1_fqdn }}"

- name: create /root/.my.cnf
  template:
    src: roles/demo_mariadb/templates/client_my_cnf.j2
    dest: "{{ mariadb_client_mycnf_path }}"
    owner: root
    group: root
    mode: 0600

- name: change root password
  mysql_user:
    name: root
    password: "{{ mariadb_root_password }}"
    priv: "{{ mariadb_root_privileges }}"
    host: "{{ item }}"
    login_host: "{{ db1_fqdn }}"
  loop:
    - 127.0.0.1
    - ::1
    - localhost

- name: restart mariadb service
  systemd:
    name: mariadb
    state: restarted
  when: mariadb_started is success and (mariadb_server_mycnf_changed is changed or mariadb_server_conf_changed is changed)
  
- name: create application database
  mysql_db:
    login_user: root
    login_password: "{{ mariadb_root_password }}"
    login_host: "{{ db1_fqdn }}"
    db: "{{ mariadb_example_database }}"
    state: present

- name: create user for application database
  mysql_user:
    login_user: root
    login_password: "{{ mariadb_root_password }}"
    login_host: "{{ db1_fqdn }}"
    user: "{{ mariadb_example_user }}"
    password: "{{ mariadb_example_password }}"
    priv: "{{ mariadb_example_privileges }}"
    state: present
    host: "{{ item }}"
  loop:
    - "localhost"
    - "%"

To test the set-up with the application presented here, we need to configure a data source in WebLogic that is either XA-enabled or using Logging Last Resource (more information can be found here). For the data source we can use the following:

[oracle@java1 jdbc]$ cat mariadb_ds-jdbc.xml
<jdbc-data-source ...>
  <name>mariadb_ds</name>
  <datasource-type>GENERIC</datasource-type>
  <jdbc-driver-params>
    <url>jdbc:mariadb://db1.ms.com:3306/exampledb</url>
    <driver-name>org.mariadb.jdbc.MariaDbPoolDataSource</driver-name>
    <properties>
      <property>
        <name>user</name>
        <value>example</value>
      </property>
    </properties>
    <password-encrypted>{AES}Q810oPbAbLfIre6q2eCbRgy4834n3Syer05DPhnYtVI=</password-encrypted>
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-table-name>SQL SELECT 1</test-table-name>
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>jdbc/exampleDS</jndi-name>
    <global-transactions-protocol>TwoPhaseCommit</global-transactions-protocol>
  </jdbc-data-source-params>
</jdbc-data-source>

Unfortunately:

ServletContext@1777542971[app:javaee7 module:javaee7 path:null spec-version:3.1]] Servlet failed with an Exception
javax.ejb.EJBException: BEA1-0006166BC1D3178E01FA; nested exception is:
        javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.5.v20170607-b3d05bd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_INVAL start() failed on resource 'mariadb_ds_tryout_domain': XAER_INVAL : Invalid arguments were given
javax.transaction.xa.XAException
        at org.mariadb.jdbc.MariaXaResource.mapXaException(MariaXaResource.java:120)
        at org.mariadb.jdbc.MariaXaResource.execute(MariaXaResource.java:136)
        at org.mariadb.jdbc.MariaXaResource.start(MariaXaResource.java:319)
        at weblogic.jdbc.jta.DataSource.start(DataSource.java:835)
        at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1395)
        at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1326)
        at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:313)
        at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:678)
        at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:497)
        at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1778)
        at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1680)
        at weblogic.jdbc.jta.DataSource.getConnectionInternal(DataSource.java:520)
        at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:497)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:614)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:594)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:587)
        at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:135)
        at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162

A bit of Searching let to this:

XA support is limited, see here and here for more information.
Known problems with mysql bug:
Some users have reported problems with MySQL XA (related to this MySQL bug). This problem only happens if you access the same MySQL database more than once in the same transaction. A workaround can be setting the following property in jta.properties: com.atomikos.icatch.serial_jta_transactions=false. Also, make sure to set the following property on the MySQL datasource: pinGlobalTxToPhysicalConnection=true MariaDB's java driver also supports this workaround since v.1.1.8.

As with so many workarounds sometimes they work sometimes they do not. With the org.mariadb.jdbc.MariaDbDataSource (mentioned here) we run into the same problems. Note that autocommit is turned off:

MariaDB [(none)]> SHOW VARIABLES WHERE Variable_name='autocommit';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| autocommit    | OFF   |
+---------------+-------+
1 row in set (0.001 sec)

MariaDB [(none)]> SELECT @@autocommit;
+--------------+
| @@autocommit |
+--------------+
|            0 |
+--------------+
1 row in set (0.000 sec)

Changing the driver to org.mariadb.jdbc.Driver and changing the global-transactions-protocol to LoggingLastResource works. In the case of LoggingLastResource WebLogic adds extra tables to the schema:

MariaDB [(none)]> use exampledb
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

MariaDB [exampledb]> show tables;
+-----------------------------+
| Tables_in_exampledb         |
+-----------------------------+
| FILM                        |
| PERSOON                     |
| WL_LLR_APPLICATION_SERVER_1 |
| WL_LLR_APPLICATION_SERVER_2 |
+-----------------------------+
4 rows in set (0.000 sec)

MariaDB [exampledb]> describe WL_LLR_APPLICATION_SERVER_1;
+-------------+---------------+------+-----+---------+-------+
| Field       | Type          | Null | Key | Default | Extra |
+-------------+---------------+------+-----+---------+-------+
| XIDSTR      | varchar(40)   | NO   | PRI | NULL    |       |
| POOLNAMESTR | varchar(64)   | YES  |     | NULL    |       |
| RECORDSTR   | varchar(1000) | YES  |     | NULL    |       |
+-------------+---------------+------+-----+---------+-------+
3 rows in set (0.001 sec)

MariaDB [exampledb]> select * from WL_LLR_APPLICATION_SERVER_1;
+-------------------------+-------------------------+-------------------------------------+
| XIDSTR                  | POOLNAMESTR             | RECORDSTR                           |
+-------------------------+-------------------------+-------------------------------------+
| JDBC LLR Domain//Server | JDBC LLR Domain//Server | tryout_domain//application_server_1 |
| JDBC LLR Version        | JDBC LLR Version        | 1.0                                 |
+-------------------------+-------------------------+-------------------------------------+
2 rows in set (0.000 sec)

More information about the Logging Last Resource feature can be found here:

In a global two-phase commit transaction with a Logging Last Resource participant, the WebLogic transaction manager follows these basic steps:

Calls prepare on all other (XA-compliant) transaction participants.

Inserts a commit record to a table on the Logging Last Resource participant (rather than to the file-based transaction log).

Commits the Logging Last Resource participant's local transaction (which includes both the transaction commit record insert and the application's SQL work).

Calls commit on all other transaction participants.

After the transaction completes successfully, lazily deletes the database transaction log entry as part of a future transaction.

PostgreSQL

To set-up the PostgreSQL database we can use the following steps:

Used variables:

postgresql_packages:
  - postgresql11
  - postgresql11-server

postgresql_bin_dir: "/usr/pgsql-11/bin"
postgresql_data_dir: "/var/lib/pgsql/11/data"
postgresql_initdb_command: "{{ postgresql_bin_dir }}/postgresql-11-setup initdb"
postgresql_user: "postgres"
postgresql_group: "postgres"
postgresql_listen_addresses: "localhost,{{ db1_fqdn }}"

postgresql_postgres_conf_path: "{{ postgresql_data_dir }}/postgresql.conf"
postgresql_pg_hba_conf_path: "{{ postgresql_data_dir }}/pg_hba.conf"

postgresql_example_database: "exampledb"
postgresql_example_user: "example"
postgresql_example_password: "set_in_inventory"
postgresql_example_privileges: "ALL"

Add the PostgreSQL repository (which can be obtained here).
Install the PostgreSQL packages.
Initialize the database.
Create an application database by using the postgresql_db module.
Create an application user by using the postgresql_user module.

Edit the PostgreSQL configuration by setting parameters, such as listen_addresses, max_connections, and max_prepared_transactions.

...
#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

listen_addresses = '{{ postgresql_listen_addresses }}'
...

Edit the pg_hba.conf file

...
# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            ident
# IPv6 local connections:
host    all             all             ::1/128                 ident
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            ident
host    replication     all             ::1/128                 ident

# Allow user example from any host with IP address {{ demo_domain_ips }} to connect to database exampledb.
#
# TYPE  DATABASE        USER            ADDRESS                 METHOD
host    {{ postgresql_example_database }}       {{ postgresql_example_user }}         {{ demo_domain_ips }}          password

To combine the above in an Ansible playbook, we have the following:

- name: install postgresql repository configuration
  yum:
    name=https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-oraclelinux11-11-2.noarch.rpm
    state=present

- name: install postgresql
  yum:
    name: "{{ postgresql_packages }}"
    state: present

#- name: install postgresql python library
#  pip:
#    name: psycopg2
#    state: latest
#  environment:
#    PATH: "{{ ansible_env.PATH }}:{{ postgresql_bin_dir }}"

- name: initialize postgresql database
  command: /bin/sh -c "{{ postgresql_initdb_command }}"
    creates="{{ postgresql_data_dir }}/PG_VERSION"
  register: postgresql_database_initialized
  
- name: enable and start postgresql database
  systemd:
    name: postgresql-11
    state: started
    enabled: yes
  register: postgresql_started
  
- name: create application database
  become: true
  become_user: "{{ postgresql_user }}"
  become_method: sudo
  postgresql_db:
    name: "{{ postgresql_example_database }}"
    state: present

- name: create user for application database
  become: true
  become_user: "{{ postgresql_user }}"
  become_method: sudo
  postgresql_user:
    name: "{{ postgresql_example_user }}"
    password: "{{ postgresql_example_password }}"
    db: "{{ postgresql_example_database }}"
    priv: "{{ postgresql_example_privileges }}" 
    state: present
    
- name: edit postgresql configuration
  template:
    src: roles/demo_postgresql/templates/postgres_conf.j2
    dest: "{{ postgresql_postgres_conf_path }}"
    owner: "{{ postgresql_user }}"
    group: "{{ postgresql_group }}"
    mode: 0600
  register: postgresql_configuration_edited
  
- name: edit postgresql client authentication
  template:
    src: roles/demo_postgresql/templates/pg_hba_conf.j2
    dest: "{{ postgresql_pg_hba_conf_path }}"
    owner: "{{ postgresql_user }}"
    group: "{{ postgresql_group }}"
    mode: 0600
  register: postgresql_client_authentication_edited
  
- name: restart postgresql database
  systemd:
    name: postgresql-11
    state: restarted
  when: postgresql_configuration_edited is changed or postgresql_client_authentication_edited is changed

[oracle@java1 jdbc]$ cat postgresql_ds-jdbc.xml
<jdbc-data-source ...>
  <name>postgresql_ds</name>
  <datasource-type>GENERIC</datasource-type>
  <jdbc-driver-params>
    <url>jdbc:postgresql://db1.ms.com:5432/exampledb</url>
    <driver-name>org.postgresql.xa.PGXADataSource</driver-name>
    <properties>
      <property>
        <name>user</name>
        <value>example</value>
      </property>
    </properties>
    <password-encrypted>{AES}E/473USUywhmwElhAOeSFnfccWfzH5T5+tg3ZAZolxM=</password-encrypted>
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-table-name>SQL SELECT 1</test-table-name>
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>jdbc/exampleDS</jndi-name>
    <global-transactions-protocol>TwoPhaseCommit</global-transactions-protocol>
  </jdbc-data-source-params>
</jdbc-data-source>

In this case everything works as expected.

On the other hand when using the org.postgresql.Driver, with the global-transactions-protocol set to LoggingLastResource does not work. In this case we sometimes get

Exception occurred during commit of transaction Name=[EJB model.logic.CompanyBean.insertPerson(model.entities.Person)],Xid=BEA1-00021F3C2813178E01FA(433518669),Status=Rolled back. [Reason=javax.transaction.SystemException: NonXAResource 'WL_JDBC_LLR_POOL.postgresql_ds' commit failed: javax.transaction.SystemException: weblogic.transaction.loggingresource.LoggingResourceException: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(1000)],numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=0,seconds left=30,useSecure=false,NonXAServerResourceInfo[WL_JDBC_LLR_POOL.postgresql_ds]=(ServerResourceInfo[WL_JDBC_LLR_POOL.postgresql_ds]=(state=rolledback,assigned=application_server_1),nonXAResource=JDBC LLR, pool=postgresql_ds, table=wl_llr_application_server_1),XAServerResourceInfo[WLStore_tryout_domain_application_cluster_filestore@application_server_2]=(ServerResourceInfo[WLStore_tryout_domain_application_cluster_filestore@application_server_2]=(state=rolledback,assigned=application_server_2),xar=null,re-Registered = false),SCInfo[tryout_domain+application_server_2]=(state=rolledback),SCInfo[tryout_domain+application_server_1]=(state=rolledback),properties=({weblogic.transaction.name=[EJB model.logic.CompanyBean.insertPerson(model.entities.Person)], weblogic.transaction.partitionName=DOMAIN, weblogic.transaction.allResourcesAssigned=true, weblogic.jdbc.remote.postgresql_ds=t3://172.20.40.144:9002, weblogic.transaction.nonXAResource=WL_JDBC_LLR_POOL.postgresql_ds, weblogic.jdbc.llr=postgresql_ds}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=application_server_1+172.20.40.144:9002+tryout_domain+t3+ CoordinatorNonSecureURL=application_server_1+172.20.40.144:9002+tryout_domain+t3+ coordinatorSecureURL=null, XAResources={WSATGatewayRM_application_server_1_tryout_domain, WLStore_tryout_domain_application_cluster_filestore@application_server_1},NonXAResources={})],CoordinatorURL=application_server_1+172.20.40.144:9002+tryout_domain+t3+): weblogic.transaction.RollbackException: NonXAResource 'WL_JDBC_LLR_POOL.postgresql_ds' commit failed: javax.transaction.SystemException: weblogic.transaction.loggingresource.LoggingResourceException: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(1000)
        at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:2104)
        at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:376)
        at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:262)
        at weblogic.ejb.container.internal.BaseRemoteObject.postInvoke1(BaseRemoteObject.java:442)
        at weblogic.ejb.container.internal.StatelessRemoteObject.postInvoke1(StatelessRemoteObject.java:20)
        at weblogic.ejb.container.internal.BaseRemoteObject.__WL_postInvokeTxRetry(BaseRemoteObject.java:307)
        at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invokeInternal(SessionRemoteMethodInvoker.java:67)
        at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:21)
        at model.logic.Company_hufak0_CompanyImpl.insertPerson(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:86)
        at com.sun.proxy.$Proxy150.insertPerson(Unknown Source)
        at userinterface.servlets.NormalServlet.doGet(NormalServlet.java:53)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

To see what is happening, we can take a look at the created tables:

-bash-4.2$ psql -h db1.ms.com -p 5432 -U example exampledb
Password for user example:
psql (11.2)
Type "help" for help.

exampledb=> \dt
                   List of relations
 Schema |            Name             | Type  |  Owner
--------+-----------------------------+-------+---------
 public | film                        | table | example
 public | persoon                     | table | example
 public | wl_llr_application_server_1 | table | example
 public | wl_llr_application_server_2 | table | example
(4 rows)
 
exampledb=> \d+ wl_llr_application_server_1
                                   Table "public.wl_llr_application_server_1"
   Column    |          Type           | Collation | Nullable | Default | Storage  | Stats target | Description
-------------+-------------------------+-----------+----------+---------+----------+--------------+-------------
 xidstr      | character varying(40)   |           | not null |         | extended |              |
 poolnamestr | character varying(64)   |           |          |         | extended |              |
 recordstr   | character varying(1000) |           |          |         | extended |              |
Indexes:
    "wl_llr_application_server_1_pkey" PRIMARY KEY, btree (xidstr)
    
exampledb=> select * from wl_llr_application_server_1;
         xidstr          |       poolnamestr       |              recordstr
-------------------------+-------------------------+-------------------------------------
 JDBC LLR Domain//Server | JDBC LLR Domain//Server | tryout_domain//application_server_1
 JDBC LLR Version        | JDBC LLR Version        | 1.0
(2 rows)

exampledb=> select * from wl_llr_application_server_2;
         xidstr          |       poolnamestr       |              recordstr
-------------------------+-------------------------+-------------------------------------
 JDBC LLR Domain//Server | JDBC LLR Domain//Server | tryout_domain//application_server_2
 JDBC LLR Version        | JDBC LLR Version        | 1.0
(2 rows)

Of course, the expection - org.postgresql.util.PSQLException: ERROR: value too long for type character varying(1000) - is clear and the column recordstr is too small. To change this we have a look at the documentation on character types. Let just choose text as this has unlimited length (note this is just a test). To change the data type of the column recordstr, we can use alter table

exampledb=> ALTER TABLE wl_llr_application_server_1 ALTER COLUMN recordstr TYPE TEXT USING recordstr :: TEXT;
ALTER TABLE

exampledb=> ALTER TABLE wl_llr_application_server_2 ALTER COLUMN recordstr TYPE TEXT USING recordstr :: TEXT;
ALTER TABLE

exampledb=> \d+ wl_llr_application_server_1
                                  Table "public.wl_llr_application_server_1"
   Column    |         Type          | Collation | Nullable | Default | Storage  | Stats target | Description
-------------+-----------------------+-----------+----------+---------+----------+--------------+-------------
 xidstr      | character varying(40) |           | not null |         | extended |              |
 poolnamestr | character varying(64) |           |          |         | extended |              |
 recordstr   | text                  |           |          |         | extended |              |
Indexes:
    "wl_llr_application_server_1_pkey" PRIMARY KEY, btree (xidstr)
    
exampledb=> select recordstr, char_length(recordstr) from wl_llr_application_server_1;
                                  recordstr                                   | char_length
------------------------------------------------------------------------------+-------------
 tryout_domain//application_server_1                                          |          35
 1.0                                                                          |           3
 rO0ABXcIAAAAWgAAAANzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0+|        1010
 b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAMdwgAAAAQAAAABnQAGXdlYmxvZ2ljLnRyYW5zYWN0aW9u+|
 Lm5hbWV0AEFbRUpCIG1vZGVsLmxvZ2ljLkNvbXBhbnlCZWFuLmluc2VydFBlcnNvbihtb2RlbC5l+|
 bnRpdGllcy5QZXJzb24pXXQAIndlYmxvZ2ljLnRyYW5zYWN0aW9uLnBhcnRpdGlvbk5hbWV0AAZE+|
 T01BSU50ACl3ZWJsb2dpYy50cmFuc2FjdGlvbi5hbGxSZXNvdXJjZXNBc3NpZ25lZHNyABFqYXZh+|
 LmxhbmcuQm9vbGVhbs0gcoDVnPruAgABWgAFdmFsdWV4cAF0ACJ3ZWJsb2dpYy5qZGJjLnJlbW90+|
 ZS5wb3N0Z3Jlc3FsX2RzdAAXdDM6Ly8xNzIuMjAuNDAuMTQ0OjkwMDJ0ACJ3ZWJsb2dpYy50cmFu+|
 c2FjdGlvbi5ub25YQVJlc291cmNldAAeV0xfSkRCQ19MTFJfUE9PTC5wb3N0Z3Jlc3FsX2RzdAAR+|
 d2VibG9naWMuamRiYy5sbHJ0AA1wb3N0Z3Jlc3FsX2RzeHf+AAAACgAVHzwoExeOAfoAAAFp82z/+|
 zgAAAAIAHldMX0pEQkNfTExSX1BPT0wucG9zdGdyZXNxbF9kcwBIV0xTdG9yZV90cnlvdXRfZG9t+|
 YWluX2FwcGxpY2F0aW9uX2NsdXN0ZXJfZmlsZXN0b3JlQGFwcGxpY2F0aW9uX3NlcnZlcl8yAAAA+|
 AgA5YXBwbGljYXRpb25fc2VydmVyXzErMTcyLjIwLjQwLjE0NDo5MDAyK3RyeW91dF9kb21haW4r+|
 dDMrADlhcHBsaWNhdGlvbl9zZXJ2ZXJfMisxNzIuMjAuNDAuMTQ2OjkwMDMrdHJ5b3V0X2RvbWFp+|
 bit0Mys=                                                                    +|
                                                                              |
 rO0ABXcIAAAAWgAAAANzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0+|         811
 b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAGdwgAAAAIAAAABXQAIndlYmxvZ2ljLmpkYmMucmVtb3Rl+|
 LnBvc3RncmVzcWxfZHN0ABd0MzovLzE3Mi4yMC40MC4xNDQ6OTAwMnQAGXdlYmxvZ2ljLnRyYW5z+|
 YWN0aW9uLm5hbWV0AEFbRUpCIG1vZGVsLmxvZ2ljLkNvbXBhbnlCZWFuLmluc2VydFBlcnNvbiht+|
 b2RlbC5lbnRpdGllcy5QZXJzb24pXXQAIndlYmxvZ2ljLnRyYW5zYWN0aW9uLm5vblhBUmVzb3Vy+|
 Y2V0AB5XTF9KREJDX0xMUl9QT09MLnBvc3RncmVzcWxfZHN0ACJ3ZWJsb2dpYy50cmFuc2FjdGlv+|
 bi5wYXJ0aXRpb25OYW1ldAAGRE9NQUlOdAARd2VibG9naWMuamRiYy5sbHJ0AA1wb3N0Z3Jlc3Fs+|
 X2RzeHfDAAAACgAWHzwoExeOAfoAAAFp820DBgAAAAIAHldMX0pEQkNfTExSX1BPT0wucG9zdGdy+|
 ZXNxbF9kcwBIV0xTdG9yZV90cnlvdXRfZG9tYWluX2FwcGxpY2F0aW9uX2NsdXN0ZXJfZmlsZXN0+|
 b3JlQGFwcGxpY2F0aW9uX3NlcnZlcl8xAAAAAQA5YXBwbGljYXRpb25fc2VydmVyXzErMTcyLjIw+|
 LjQwLjE0NDo5MDAyK3RyeW91dF9kb21haW4rdDMr                                    +|
                                                                              |
 rO0ABXcIAAAAWgAAAANzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0+|        1010
 b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAMdwgAAAAQAAAABnQAGXdlYmxvZ2ljLnRyYW5zYWN0aW9u+|
 Lm5hbWV0AEFbRUpCIG1vZGVsLmxvZ2ljLkNvbXBhbnlCZWFuLmluc2VydFBlcnNvbihtb2RlbC5l+|
 bnRpdGllcy5QZXJzb24pXXQAIndlYmxvZ2ljLnRyYW5zYWN0aW9uLnBhcnRpdGlvbk5hbWV0AAZE+|
 T01BSU50ACl3ZWJsb2dpYy50cmFuc2FjdGlvbi5hbGxSZXNvdXJjZXNBc3NpZ25lZHNyABFqYXZh+|
 LmxhbmcuQm9vbGVhbs0gcoDVnPruAgABWgAFdmFsdWV4cAF0ACJ3ZWJsb2dpYy5qZGJjLnJlbW90+|
 ZS5wb3N0Z3Jlc3FsX2RzdAAXdDM6Ly8xNzIuMjAuNDAuMTQ0OjkwMDJ0ACJ3ZWJsb2dpYy50cmFu+|
 c2FjdGlvbi5ub25YQVJlc291cmNldAAeV0xfSkRCQ19MTFJfUE9PTC5wb3N0Z3Jlc3FsX2RzdAAR+|
 d2VibG9naWMuamRiYy5sbHJ0AA1wb3N0Z3Jlc3FsX2RzeHf+AAAACgAZHzwoExeOAfoAAAFp820G+|
 LAAAAAIAHldMX0pEQkNfTExSX1BPT0wucG9zdGdyZXNxbF9kcwBIV0xTdG9yZV90cnlvdXRfZG9t+|
 YWluX2FwcGxpY2F0aW9uX2NsdXN0ZXJfZmlsZXN0b3JlQGFwcGxpY2F0aW9uX3NlcnZlcl8yAAAA+|
 AgA5YXBwbGljYXRpb25fc2VydmVyXzErMTcyLjIwLjQwLjE0NDo5MDAyK3RyeW91dF9kb21haW4r+|
 dDMrADlhcHBsaWNhdGlvbl9zZXJ2ZXJfMisxNzIuMjAuNDAuMTQ2OjkwMDMrdHJ5b3V0X2RvbWFp+|
 bit0Mys=                                                                    +|
                                                                              |
 rO0ABXcIAAAAWgAAAANzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0+|         811
 b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAGdwgAAAAIAAAABXQAIndlYmxvZ2ljLmpkYmMucmVtb3Rl+|
 LnBvc3RncmVzcWxfZHN0ABd0MzovLzE3Mi4yMC40MC4xNDQ6OTAwMnQAGXdlYmxvZ2ljLnRyYW5z+|
 YWN0aW9uLm5hbWV0AEFbRUpCIG1vZGVsLmxvZ2ljLkNvbXBhbnlCZWFuLmluc2VydFBlcnNvbiht+|
 b2RlbC5lbnRpdGllcy5QZXJzb24pXXQAIndlYmxvZ2ljLnRyYW5zYWN0aW9uLm5vblhBUmVzb3Vy+|
 Y2V0AB5XTF9KREJDX0xMUl9QT09MLnBvc3RncmVzcWxfZHN0ACJ3ZWJsb2dpYy50cmFuc2FjdGlv+|
 bi5wYXJ0aXRpb25OYW1ldAAGRE9NQUlOdAARd2VibG9naWMuamRiYy5sbHJ0AA1wb3N0Z3Jlc3Fs+|
 X2RzeHfDAAAACgAaHzwoExeOAfoAAAFp820JNAAAAAIAHldMX0pEQkNfTExSX1BPT0wucG9zdGdy+|
 ZXNxbF9kcwBIV0xTdG9yZV90cnlvdXRfZG9tYWluX2FwcGxpY2F0aW9uX2NsdXN0ZXJfZmlsZXN0+|
 b3JlQGFwcGxpY2F0aW9uX3NlcnZlcl8xAAAAAQA5YXBwbGljYXRpb25fc2VydmVyXzErMTcyLjIw+|
 LjQwLjE0NDo5MDAyK3RyeW91dF9kb21haW4rdDMr                                    +|
                                                                              |
 rO0ABXcIAAAAWgAAAANzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0+|        1010
 b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAMdwgAAAAQAAAABnQAGXdlYmxvZ2ljLnRyYW5zYWN0aW9u+|
 Lm5hbWV0AEFbRUpCIG1vZGVsLmxvZ2ljLkNvbXBhbnlCZWFuLmluc2VydFBlcnNvbihtb2RlbC5l+|
 bnRpdGllcy5QZXJzb24pXXQAIndlYmxvZ2ljLnRyYW5zYWN0aW9uLnBhcnRpdGlvbk5hbWV0AAZE+|
 T01BSU50ACl3ZWJsb2dpYy50cmFuc2FjdGlvbi5hbGxSZXNvdXJjZXNBc3NpZ25lZHNyABFqYXZh+|
 LmxhbmcuQm9vbGVhbs0gcoDVnPruAgABWgAFdmFsdWV4cAF0ACJ3ZWJsb2dpYy5qZGJjLnJlbW90+|
 ZS5wb3N0Z3Jlc3FsX2RzdAAXdDM6Ly8xNzIuMjAuNDAuMTQ0OjkwMDJ0ACJ3ZWJsb2dpYy50cmFu+|
 c2FjdGlvbi5ub25YQVJlc291cmNldAAeV0xfSkRCQ19MTFJfUE9PTC5wb3N0Z3Jlc3FsX2RzdAAR+|
 d2VibG9naWMuamRiYy5sbHJ0AA1wb3N0Z3Jlc3FsX2RzeHf+AAAACgAcHzwoExeOAfoAAAFp820M+|
 XgAAAAIAHldMX0pEQkNfTExSX1BPT0wucG9zdGdyZXNxbF9kcwBIV0xTdG9yZV90cnlvdXRfZG9t+|
 YWluX2FwcGxpY2F0aW9uX2NsdXN0ZXJfZmlsZXN0b3JlQGFwcGxpY2F0aW9uX3NlcnZlcl8yAAAA+|
 AgA5YXBwbGljYXRpb25fc2VydmVyXzErMTcyLjIwLjQwLjE0NDo5MDAyK3RyeW91dF9kb21haW4r+|
 dDMrADlhcHBsaWNhdGlvbl9zZXJ2ZXJfMisxNzIuMjAuNDAuMTQ2OjkwMDMrdHJ5b3V0X2RvbWFp+|
 bit0Mys=                                                                    +|
                                                                              |
 rO0ABXcIAAAAWgAAAANzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0+|         811
 b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAGdwgAAAAIAAAABXQAIndlYmxvZ2ljLmpkYmMucmVtb3Rl+|
 LnBvc3RncmVzcWxfZHN0ABd0MzovLzE3Mi4yMC40MC4xNDQ6OTAwMnQAGXdlYmxvZ2ljLnRyYW5z+|
 YWN0aW9uLm5hbWV0AEFbRUpCIG1vZGVsLmxvZ2ljLkNvbXBhbnlCZWFuLmluc2VydFBlcnNvbiht+|
 b2RlbC5lbnRpdGllcy5QZXJzb24pXXQAIndlYmxvZ2ljLnRyYW5zYWN0aW9uLm5vblhBUmVzb3Vy+|
 Y2V0AB5XTF9KREJDX0xMUl9QT09MLnBvc3RncmVzcWxfZHN0ACJ3ZWJsb2dpYy50cmFuc2FjdGlv+|
 bi5wYXJ0aXRpb25OYW1ldAAGRE9NQUlOdAARd2VibG9naWMuamRiYy5sbHJ0AA1wb3N0Z3Jlc3Fs+|
 X2RzeHfDAAAACgAeHzwoExeOAfoAAAFp820TqwAAAAIAHldMX0pEQkNfTExSX1BPT0wucG9zdGdy+|
 ZXNxbF9kcwBIV0xTdG9yZV90cnlvdXRfZG9tYWluX2FwcGxpY2F0aW9uX2NsdXN0ZXJfZmlsZXN0+|
 b3JlQGFwcGxpY2F0aW9uX3NlcnZlcl8xAAAAAQA5YXBwbGljYXRpb25fc2VydmVyXzErMTcyLjIw+|
 LjQwLjE0NDo5MDAyK3RyeW91dF9kb21haW4rdDMr                                    +|
                                                                              |
(8 rows)

Above, we see that indeed sometimes a record with a length of 1010 is inserted.

Directory Server

To set-up the Directory Server we can use the following steps:

Used variables:

ds_packages:
  - 389-ds-base
  - 389-admin
  #- 389-console
  - 389-ds-console
  - 389-admin-console

ds_silent_setup_path: "/tmp/setup.inf"
ds_nslcd_conf_path: "/etc/nslcd.conf"

# https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Installation_Guide/Installation_Guide-Advanced_Configuration-Silent.html
ds_full_machine_name: "{{ db1_fqdn }}"
ds_system_user: "dirsrv"
ds_system_group: "dirsrv"
ds_administrator_id : "admin"
ds_administrator_password: "{{ overall_password }}"
ds_administration_domain: "{{ demo_domain_name }}"

ds_directory_server_network_port: "389"
ds_directory_server_identifier: "{{ db1_hostname }}"
ds_suffix: "dc=ms,dc=com"
ds_directory_manager_dn: "cn=Directory Manager"
ds_directory_manager_password: "{{ overall_password }}"

ds_administration_port: "9830"

ds_ldap_authentication:
  server_uri: "ldap://{{ ds_full_machine_name }}:{{ ds_directory_server_network_port }}"
  bind_dn: "{{ ds_directory_manager_dn }}"
  bind_pw: "{{ overall_password }}"
  
ds_users_dn: "ou=People,{{ ds_suffix }}"
ds_groups_dn: "ou=Groups,{{ ds_suffix }}"

Install the 389 Directory Server packages

Create setup file for the setup-ds-admin.pl script

[General] 
FullMachineName={{ ds_full_machine_name }}
SuiteSpotUserID={{ ds_system_user }}
SuiteSpotGroup={{ ds_system_group }}
ConfigDirectoryAdminID={{ ds_administrator_id }}
ConfigDirectoryAdminPwd={{ ds_administrator_password }}
AdminDomain={{ ds_administration_domain }}

[slapd] 
ServerPort={{ ds_directory_server_network_port }}
ServerIdentifier={{ ds_directory_server_identifier }}
Suffix={{ ds_suffix }}
RootDN={{ ds_directory_manager_dn }}
RootDNPwd={{ ds_directory_manager_password }}

[admin] 
Port={{ ds_administration_port }}

More information about the silent setup can be found here.

Execute setup-ds.pl or setup-ds-admin.pl.
Enable and start the Directory Server and the Administration Server.

Add openssh ldap schema, such that we can add ssh keys to users, more information can be found here

# openssh-ldap schema for 389
dn: cn=schema
#
attributetypes: ( 1.3.6.1.4.1.24552.500.1.1.1.13
        NAME 'sshPublicKey'
        DESC 'MANDATORY: OpenSSH Public key'
        EQUALITY octetStringMatch
        SYNTAX 1.3.6.1.4.1.1466.115.121.1.40
        )
#
objectclasses: ( 1.3.6.1.4.1.24552.500.1.1.2.0
        NAME 'ldapPublicKey' SUP top AUXILIARY
        DESC 'MANDATORY: OpenSSH LPK objectclass'
        MUST uid
        MAY sshPublicKey
        )

Restart the Directory Server.

To combine the above in an Ansible playbook, we have the following:

- name: install 389 directory server
  yum:
    name: "{{ ds_packages }}"
    state: present

- name: create temporary setup files
  template:
    src: roles/demo_directory_server/templates/setup_inf.j2
    dest: "{{ ds_silent_setup_path }}"
    owner: root
    group: root
    mode: 0644
  register: silent_setup_file_created

#- name: execute silent setup
#  command: /bin/sh -c "/usr/sbin/setup-ds-admin.pl -s -f {{ ds_silent_setup_path }}"
#    creates="/etc/dirsrv/slapd-{{ ds_directory_server_identifier }}/dse.ldif"
#  register: executed_silent_setup
#  when: silent_setup_file_created is changed
  
- name: execute silent setup
  command: /bin/sh -c "/usr/sbin/setup-ds.pl -s -f {{ ds_silent_setup_path }}"
    creates="/etc/dirsrv/slapd-{{ ds_directory_server_identifier }}/dse.ldif"
  register: executed_silent_setup
  when: silent_setup_file_created is changed

- name: debug silent setup
  debug:
    var=executed_silent_setup
  when: executed_silent_setup is changed

- name: delete temporary setup files that contain usernames and passwords
  file:
    path="{{ item }}"
    state=absent
  loop:
    - "{{ ds_silent_setup_path }}"

- name: enable and start 389 directory server
  systemd:
    name: dirsrv.target
    state: started
    enabled: yes
  register: dirsrv_started
  
- name: check admin-server directory
  stat:
    path: /var/log/dirsrv/admin-serv
  register: admin_server_directory

- name: enable and start 389 administration server
  systemd:
    name: dirsrv-admin
    state: started
    enabled: yes
  when: admin_server_directory.stat.exists
  register: dirsrv_admin_started
  
- name: add openssh ldap schema
  template:
    src: roles/demo_directory_server/templates/98openssh_ldap_ldif.j2
    dest: "/etc/dirsrv/slapd-{{ ds_directory_server_identifier }}/schema/98openssh_ldap.ldif"
    owner: "{{ ds_system_user }}"
    group: "{{ ds_system_group }}"
    mode: 0600
  register: openssh_ldap_ldif_file_created
  
- name: restart 389 directory server
  systemd:
    name: dirsrv.target
    state: restarted
  when: openssh_ldap_ldif_file_created is changed

Note that the setup-ds.pl command creates an instance of Directory Server without installing the Administration Server or Directory Server Console. It works exactly the same way as setup-ds-admin.pl, except that the questions about the Configuration Directory Server and Administration Server are omitted. Using this command to create a Directory Server instance means that the instance has to be managed through the command line.

When needed we can also configure secure connections for the Directory Server, to this end we can use the following steps:

Used variables:

ds_directory_server_secure_network_port: "636"

ds_ldaps_authentication:
  server_uri: "ldaps://{{ ds_full_machine_name }}:{{ ds_directory_server_secure_network_port }}"
  bind_dn: "{{ ds_directory_manager_dn }}"
  bind_pw: "{{ ds_directory_manager_password }}"
  
ds_nss_database_password: "password"
ds_nss_database_path: "/etc/dirsrv/slapd-{{ ds_directory_server_identifier }}"
ds_nss_database_certutil_password_file_path: "{{ ds_nss_database_path }}/pwdfile"
ds_nss_database_password_file_path: "{{ ds_nss_database_path }}/pin.txt"
ds_nss_database_server_certificate_name: "server-cert"
ds_nss_database_ca_certificate_path: "/root/{{ ds_nss_database_server_certificate_name }}.pem"
ds_nss_database_fetched_ca_certificate_directory: "/etc/ansible/roles/demo_directory_server_enable_tls/files/"
ds_nss_database_ciphers: "-all,+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,+TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,+TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA,+TLS_RSA_WITH_AES_128_CBC_SHA256,+TLS_RSA_WITH_AES_128_GCM_SHA256,+TLS_RSA_WITH_AES_256_CBC_SHA,+TLS_RSA_WITH_AES_256_CBC_SHA256,+TLS_RSA_WITH_CAMELLIA_128_CBC_SHA,+TLS_RSA_WITH_CAMELLIA_256_CBC_SHA,+TLS_RSA_WITH_SEED_CBC_SHA"

Initialize the NSS Database used by the directory server.
For testing purposes we generate a self-signed certificate.

Configure TLS in the directory server, i.e.,

dn: cn=config
changetype: modify
replace: nsslapd-securePort
nsslapd-securePort: 636
-
replace: nsslapd-security
nsslapd-security: on
-
replace: nsslapd-ssl-check-hostname
nsslapd-ssl-check-hostname: off
-
replace: nsslapd-minssf
nsslapd-minssf: 0
-
replace: nsslapd-validate-cert
nsslapd-validate-cert: warn

Configure the RSA cipher family, i.e.,

dn: cn=RSA,cn=encryption,cn=config
cn: RSA
objectClass: top
objectClass: nsEncryptionModule
nsSSLToken: internal (software)
nsSSLPersonalitySSL: {{ ds_nss_database_server_certificate_name }}
nsSSLActivation: on

Configure encryption and ciphers, i.e.,

dn: cn=encryption,cn=config
changetype: modify
add: nsSSL3Ciphers
nsSSL3Ciphers: {{ ds_nss_database_ciphers }}
-
replace: sslVersionMin
sslVersionMin: TLS1.1
-
replace: sslVersionMax
sslVersionMax: TLS1.2
-
replace: nsSSLClientAuth
nsSSLClientAuth: allowed

With the confguration in place restart the Directory Server.

Test the set-up, to this end we edit the /etc/openldap/ldap.conf file

#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

#BASE   dc=example,dc=com
#URI    ldap://ldap.example.com ldap://ldap-master.example.com:666

#SIZELIMIT      12
#TIMELIMIT      15
#DEREF          never

#TLS_CACERTDIR  /etc/openldap/certs

URI             {{ ds_ldap_authentication.server_uri }}
BASE            {{ ds_suffix }}
HOST            {{ ds_full_machine_name }}

#TLS_CACERTDIR   {{ ds_nss_database_path }}
TLS_CACERT     {{ ds_nss_database_ca_certificate_path }}
TLS_REQCERT     allow

# Turning this off breaks GSSAPI used with krb5 when rdns = false
SASL_NOCANON    on

and run a ldapsearch command that uses the ldaps protocol, i.e.,

/usr/bin/ldapsearch -H {{ ds_ldaps_authentication.server_uri }} -D '{{ ds_ldaps_authentication.bind_dn }}' -w {{ ds_ldaps_authentication.bind_pw }} -x -b 'cn=config' '(objectclass=nsslapdConfig)' 'nsslapd-security'

To combine the above in an Ansible playbook, we have the following:

# roles/demo_directory_server_enable_tls/tasks/main.yml

- name: check if nss database has been initialized
  command: /bin/sh -c "/usr/bin/certutil -d {{ ds_nss_database_path }} -L"
  register: database_init_check
  changed_when: false
  failed_when: "'Database needs user init' in database_init_check.stdout"
  ignore_errors: true
 
- name: debug check if nss database has been initialized
  debug:
    var=database_init_check
  changed_when: false
  
- name: generate password file for certutil
  template: 
    src: roles/demo_directory_server_enable_tls/templates/pwdfile.j2
    dest: "{{ ds_nss_database_certutil_password_file_path }}"
    owner: dirsrv
    group: dirsrv
    mode: 0400
  register: certutil_password_file_created
  
- name: generate password file for nss database
  template: 
    src: roles/demo_directory_server_enable_tls/templates/pin_txt.j2
    dest: "{{ ds_nss_database_password_file_path }}"
    owner: dirsrv
    group: dirsrv
    mode: 0400
  register: nss_database_password_file_created

- name: initialize nss database
  command: /bin/sh -c "/usr/bin/certutil -d {{ ds_nss_database_path }} -f {{ ds_nss_database_certutil_password_file_path }} -N"
  register: database_initialized
  when: database_init_check is failed and certutil_password_file_created is success
  
- name: edit nss database file permissions
  file:
    path: "{{ ds_nss_database_path }}/{{ item }}"
    owner: dirsrv
    group: dirsrv
    mode: 0600
  loop:
    - cert8.db
    - key3.db
    - secmod.db
    
- name: generate noise file
  command: /bin/sh -c "openssl rand -out /root/noise.bin 4096"
    creates=/root/noise.bin
  register: noise_file_generated
    
- name: check if server certificate exists
  command: /bin/sh -c "/usr/bin/certutil -d {{ ds_nss_database_path }} -L"
  register: server_certificate_check
  changed_when: false
  failed_when: "ds_nss_database_server_certificate_name not in server_certificate_check.stdout"
  ignore_errors: true
 
- name: debug check if server certificate exists
  debug:
    var=server_certificate_check
  changed_when: false

- name: create a self-signed certificate
  command: /bin/sh -c "/usr/bin/certutil -S -x -d {{ ds_nss_database_path }} -f {{ ds_nss_database_certutil_password_file_path }} -z /root/noise.bin -n '{{ ds_nss_database_server_certificate_name }}' -s 'CN=$HOSTNAME' -t 'CT,C,C' -m $RANDOM --keyUsage digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment"
  when: server_certificate_check is failed and noise_file_generated is success
  register: certificate_created
  
- name: export certificate
  command: /bin/sh -c "certutil -d /etc/dirsrv/slapd-test151/ -L -n 'server-cert' -a > {{ ds_nss_database_ca_certificate_path }}"
  register: certificate_exported
  when: certificate_created is changed
  
- name: change selinux file context for exported certificate
  command: /bin/sh -c "chcon system_u:object_r:cert_t:s0 {{ ds_nss_database_ca_certificate_path }}"
  when: certificate_exported is changed

- name: fetch exported certificate
  fetch:
    src: "{{ ds_nss_database_ca_certificate_path }}"
    dest: "{{ ds_nss_database_fetched_ca_certificate_directory }}"
    flat: yes
  when: certificate_exported is changed

# https://pagure.io/389-ds-base/issue/49390
#- name: enable tls
#  ldap_attr:
#    dn: "cn=config"
#    name: "{{ item.attribute_name }}"
#    values: "{{ item.attribute_value }}"
#    params: "{{ ds_ldap_authentication }}"
#  loop:
#    - { attribute_name: 'nsslapd-securePort', attribute_value: '636' }
#    - { attribute_name: 'nsslapd-security', attribute_value: 'on' }
#    - { attribute_name: 'nsslapd-ssl-check-hostname', attribute_value: 'off' }
#    - { attribute_name: 'nsslapd-minssf', attribute_value: '0' }
#    - { attribute_name: 'nsslapd-validate-cert', attribute_value: 'cert' }
#  register: tls_enabled
  
- name: check enable tls
  command: /bin/sh -c "/usr/bin/ldapsearch -H {{ ds_ldap_authentication.server_uri }} -D '{{ ds_ldap_authentication.bind_dn }}' -w {{ ds_ldap_authentication.bind_pw }} -x -b 'cn=config' '(objectclass=nsslapdConfig)' 'nsslapd-security'"
  register: enable_tls_check
  changed_when: false
  failed_when: "'nsslapd-security: on' not in enable_tls_check.stdout"
  ignore_errors: true

- name: debug check enable tls
  debug:
    var=enable_tls_check
  changed_when: false
  
- name: create enable tls template
  template: 
    src: roles/demo_directory_server_enable_tls/templates/configure_tls_ldif.j2
    dest: "/root/configure_tls.ldif"
  register: enable_tls_template_created

- name: enable tls
  command: /bin/sh -c "/usr/bin/ldapmodify -H {{ ds_ldap_authentication.server_uri }} -D '{{ ds_ldap_authentication.bind_dn }}' -w {{ ds_ldap_authentication.bind_pw }} -x -f /root/configure_tls.ldif"
  when: enable_tls_check is failed and enable_tls_template_created is success
  register: tls_enabled

- name: enable the rsa cipher family
  ldap_entry:
    dn: cn=RSA,cn=encryption,cn=config
    state: present
    objectClass:
      - top
      - nsEncryptionModule
    attributes:
      cn: RSA
      nsSSLToken: internal (software)
      nsSSLPersonalitySSL: "{{ ds_nss_database_server_certificate_name }}"
      nsSSLActivation: on
    params: "{{ ds_ldap_authentication }}"
  register: rsa_cipher_family_enabled

# https://pagure.io/389-ds-base/issue/49390
#- name: edit ciphers and encryption
#  ldap_attr:
#    dn: "cn=encryption,cn=config"
#    name: "{{ item.attribute_name }}"
#    values: "{{ item.attribute_value }}"
#    params: "{{ ds_ldap_authentication }}"
#  loop:
#    - { attribute_name: 'nsSSL3Ciphers', attribute_value: "{{ ds_nss_database_ciphers }}" }
#    - { attribute_name: 'sslVersionMin', attribute_value: 'TLS1.1' }
#    - { attribute_name: 'sslVersionMax', attribute_value: 'TLS1.2' }
#    - { attribute_name: 'nsSSLClientAuth', attribute_value: 'allowed' }
#  register: ciphers_and_encryption_edited

- name: check ciphers and encryption
  command: /bin/sh -c "/usr/bin/ldapsearch -H {{ ds_ldap_authentication.server_uri }} -D '{{ ds_ldap_authentication.bind_dn }}' -w {{ ds_ldap_authentication.bind_pw }} -x -b 'cn=encryption,cn=config' '(objectclass=nsEncryptionConfig)' 'nsSSL3Ciphers'"
  register: ciphers_and_encryption_check
  changed_when: false
  failed_when: "'nsSSL3Ciphers: -all' not in ciphers_and_encryption_check.stdout"
  ignore_errors: true

- name: debug check enable tls
  debug:
    var=ciphers_and_encryption_check
  changed_when: false
  
- name: create ciphers and encryption template
  template: 
    src: roles/demo_directory_server_enable_tls/templates/configure_encryption_ldif.j2
    dest: "/root/configure_encryption.ldif"
  register: ciphers_and_encryption_template_created

- name: edit ciphers and encryption
  command: /bin/sh -c "/usr/bin/ldapmodify -H {{ ds_ldap_authentication.server_uri }} -D '{{ ds_ldap_authentication.bind_dn }}' -w {{ ds_ldap_authentication.bind_pw }} -x -f /root/configure_encryption.ldif"
  when: ciphers_and_encryption_check is failed and ciphers_and_encryption_template_created is success
  register: ciphers_and_encryption_edited

- name: restart 389 directory server
  systemd:
    name: dirsrv.target
    state: restarted
  when: tls_enabled is changed or rsa_cipher_family_enabled is changed or ciphers_and_encryption_edited is changed

- name: edit /etc/openldap/ldap.conf
  template: 
    src: roles/demo_directory_server_enable_tls/templates/ldap_conf.j2
    dest: /etc/openldap/ldap.conf
    owner: root
    group: root
    mode: 0644
  register: nss_database_password_file_created
  
- name: test tls
  command: /bin/sh -c "/usr/bin/ldapsearch -H {{ ds_ldaps_authentication.server_uri }} -D '{{ ds_ldaps_authentication.bind_dn }}' -w {{ ds_ldaps_authentication.bind_pw }} -x -b 'cn=config' '(objectclass=nsslapdConfig)' 'nsslapd-security'"
  register: test_tls
  changed_when: false
  when: nss_database_password_file_created is success
  
- name: debug test tls
  debug:
    var=test_tls
  changed_when: false
  when: nss_database_password_file_created is success

WebLogic Authentication, Authorization and Role Mapping - To test the setup let us add some users that are used by WebLogic. To this end, we add some users and groups to the Directory Server. To do this we can use either direct commands, i.e., ldapadd, ldapdelete, ldapmodify, and ldapsearch, or the Ansible modules ldap_entry and ldap_attr.

To use the direct commands, we first need to create some ldif files, for example,

[root@db1 ~]# cat groups.ldif
dn: cn=managers,ou=Groups,dc=ms,dc=com
objectClass: posixGroup
objectClass: top
objectClass: groupOfUniqueNames
cn: managers
gidNumber: 1000
description: manager group

dn: cn=employees,ou=Groups,dc=ms,dc=com
objectClass: posixGroup
objectClass: top
objectClass: groupOfUniqueNames
cn: employees
gidNumber: 1001
description: employee group

dn: cn=Deployers,ou=Groups,dc=ms,dc=com
objectClass: posixGroup
objectClass: top
objectClass: groupOfUniqueNames
cn: Deployers
gidNumber: 1002
description: Deployer Group


[root@db1 ~]# cat users.ldif
dn: cn=manager,ou=People,dc=ms,dc=com
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
cn: manager
sn: manager
uid: manager
userPassword: non_hashed_password
description: a manager

dn: cn=employee,ou=People,dc=ms,dc=com
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
cn: employee
sn: employee
uid: employee
userPassword: non_hashed_password
description: an employee

dn: cn=Deployer,ou=People,dc=ms,dc=com
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
cn: Deployer
sn: Deployer
uid: Deployer
userPassword: non_hashed_password
description: A Deployer


[root@db1 ~]# cat add_users_to_groups.ldif
dn: cn=managers,ou=Groups,dc=ms,dc=com
changetype: modify
add: memberUid
memberUid: cn=manager,ou=People,dc=ms,dc=com

dn: cn=employees,ou=Groups,dc=ms,dc=com
changetype: modify
add: memberUid
memberUid: cn=employee,ou=People,dc=ms,dc=com

dn: cn=Deployers,ou=Groups,dc=ms,dc=com
changetype: modify
add: memberUid
memberUid: cn=Deployer,ou=People,dc=ms,dc=com

#dn: cn=Deployers,ou=Groups,dc=ms,dc=com
#changetype: modify
#replace: memberUid
#memberUid: cn=Deployer,ou=People,dc=ms,dc=com

Note that the passwords must not be hashed (more information can found be here):

Should I hash passwords in my LDIF files? No! You must always submit a clear text password to the directory server in an ldap add or modify request. "System" passwords like nsmultiplexorcredentials for example will be base64 encoded when viewed in the dse.ldif file. Any entry in the dse.ldif file that contains a :: after the name is base64 encoded. All "system" passwords will use the same level of encryption and this level is not changeable. User password encryption level can be changed in the 389-console or by using:
ldapmodify -x -D "cn=directory manager" -w password
dn: cn=config
changetype: modify
replace: passwordStorageScheme
passwordStorageScheme: <encryption scheme here>

Initially, the Directory Server is set-up with some groups that we want to get rid of

[root@db1 ~]# ldapdelete -D "cn=directory manager" -W -h db1.ms.com -p 389
Enter LDAP Password:
cn=Accounting Managers,ou=Groups,dc=ms,dc=com
cn=HR Managers,ou=Groups,dc=ms,dc=com
cn=QA Managers,ou=Groups,dc=ms,dc=com
cn=PD Managers,ou=Groups,dc=ms,dc=com
^C

such that we are left with

[root@db1 ~]# ldapsearch -x -b "dc=ms, dc=com"
# extended LDIF
#
# LDAPv3
# base <dc=ms, dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# ms.com
dn: dc=ms,dc=com
objectClass: top
objectClass: domain
dc: ms

# Directory Administrators, ms.com
dn: cn=Directory Administrators,dc=ms,dc=com
objectClass: top
objectClass: groupofuniquenames
cn: Directory Administrators
uniqueMember: cn=Directory Manager

# Groups, ms.com
dn: ou=Groups,dc=ms,dc=com
objectClass: top
objectClass: organizationalunit
ou: Groups

# People, ms.com
dn: ou=People,dc=ms,dc=com
objectClass: top
objectClass: organizationalunit
ou: People

# Special Users, ms.com
dn: ou=Special Users,dc=ms,dc=com
objectClass: top
objectClass: organizationalUnit
ou: Special Users
description: Special Administrative Accounts

# search result
search: 2
result: 0 Success

# numResponses: 6
# numEntries: 5

To add our users and groups we execute the following commands:

[root@db1 ~]# ldapadd -D "cn=directory manager" -W -h db1.ms.com -p 389 -x -f groups.ldif
Enter LDAP Password:
adding new entry "cn=managers,ou=Groups,dc=ms,dc=com"
adding new entry "cn=employees,ou=Groups,dc=ms,dc=com"
adding new entry "cn=Deployers,ou=Groups,dc=ms,dc=com"

[root@db1 ~]# ldapadd -D "cn=directory manager" -W -h db1.ms.com -p 389 -x -f users.ldif
Enter LDAP Password:
adding new entry "cn=manager,ou=People,dc=ms,dc=com"
adding new entry "cn=employee,ou=People,dc=ms,dc=com"
adding new entry "cn=Deployer,ou=People,dc=ms,dc=com"

[root@db1 ~]# ldapmodify -D "cn=directory manager" -W -h db1.ms.com -p 389 -x -f add_users_to_groups.ldif
Enter LDAP Password:
modifying entry "cn=managers,ou=Groups,dc=ms,dc=com"
modifying entry "cn=employees,ou=Groups,dc=ms,dc=com"
modifying entry "cn=Deployers,ou=Groups,dc=ms,dc=com"

[root@db1 ~]# ldapsearch -D "cn=directory manager" -W -h db1.ms.com -p 389 -b "cn=managers,ou=Groups,dc=ms,dc=com"
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <cn=managers,ou=Groups,dc=ms,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# managers, Groups, ms.com
dn: cn=managers,ou=Groups,dc=ms,dc=com
cn: managers
objectClass: posixGroup
objectClass: top
objectClass: groupOfUniqueNames
memberUid: cn=manager,ou=People,dc=ms,dc=com
gidNumber: 1000
description: managers group

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

[root@db1 ~]# ldapsearch -D "cn=directory manager" -W -h db1.ms.com -p 389 -b "cn=manager,ou=People,dc=ms,dc=com"
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <cn=manager,ou=People,dc=ms,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# manager, People, ms.com
dn: cn=manager,ou=People,dc=ms,dc=com
description: a manager
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
sn: manager
uid: manager
cn: manager
userPassword:: e1NTSEE1MTJ9MTczSnFjY0NqUWVMRTQrRitxM0pEc0hsOHhveTRjaWNNSUNnZnF
 Dc2ZZQ2krYnpoYTRVQkdhS3JrN1NYRWV0NHh5Mjg5UUZCUHpZcWhwRC9qQWlPREVBazFma1hxTDdy

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

With the groups and users in place, we continue with configuring WebLogic (how to configure an Authentication Provider is descibed here):

WebLogic Config:
----- Common
Control Flag: SUFFICIENT
----- Provider Specific
--- Connnection
Host: db1.ms.com
Port: 389
Principal: cn=Directory Manager
Credential: password
Confirm Credential: password
--- Users
User Base DN: ou=people, dc=ms, dc=com
All Users Filter: (&(cn=*)(objectclass=person))
User From Name Filter: (&(cn=%u)(objectclass=person))
User Search Scope: subtree
User Name Attribute: cn
User Object Class: person
Use Retrieved User Name as Principal: checked
--- Groups
Group Base DN: ou=groups, dc=ms, dc=com
All Groups Filter: (&(cn=*)(objectclass=groupOfUniqueNames))
Group From Name Filter: (&(cn=%g)(objectclass=groupOfUniqueNames))
Group Search Scope: subtree
Group Membership Searching: unlimited
Max Group Membership Search Level: 0
--- Static Groups
Static Group Name Attribute: cn
Static Group Object Class: groupOfUniqueNames
Static Member DN Attribute: memberUid
Static Group DNs from Member DN Filter: (&(memberUid=%M)(objectclass=groupOfUniqueNames))

If needed we can re-order the authentication providers. On the Users and Groups tab, we can check if the configuration is correct.

To test the setup create a JavaEE application that has a security config such as:

# web.xml
<web-app ...>
    <servlet>
        <servlet-name>TestServlet</servlet-name>
        <servlet-class>userinterface.servlets.TestServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>TestServlet</servlet-name>
        <url-pattern>/test</url-pattern>
    </servlet-mapping>
    <security-constraint>
        <web-resource-collection>
            <web-resource-name>All</web-resource-name>
            <url-pattern>/test</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <role-name>MANAGER</role-name>
            <role-name>EMPLOYEE</role-name>
        </auth-constraint>
        <user-data-constraint>
            <transport-guarantee>NONE</transport-guarantee>
        </user-data-constraint>
    </security-constraint>
    <login-config>
        <auth-method>FORM</auth-method>
        <form-login-config>
            <form-login-page>/login.jsp</form-login-page>
            <form-error-page>/login.jsp</form-error-page>
        </form-login-config>
    </login-config>
    <security-role>
        <role-name>MANAGER</role-name>
    </security-role>
    <security-role>
        <role-name>EMPLOYEE</role-name>
    </security-role>
</web-app>

# weblogic.xml
<weblogic-web-app ...>
  <security-role-assignment>
    <role-name>EMPLOYEE</role-name>
    <principal-name>employees</principal-name>
  </security-role-assignment>
  <security-role-assignment>
    <role-name>MANAGER</role-name>
    <principal-name>managers</principal-name>
  </security-role-assignment>
  <session-descriptor>
    <timeout-secs>120</timeout-secs>
    <invalidation-interval-secs>30</invalidation-interval-secs>
    <persistent-store-type>replicated_if_clustered</persistent-store-type>
  </session-descriptor>
</weblogic-web-app>

or simply login to WebLogic Admin Console using the Deployer user. By tailing the Directory Server's access log we can see what is going on:

[root@db1 ~]# tail -f /var/log/dirsrv/slapd-db1/access
[07/Apr/2019:04:50:15.998788675 -0400] conn=73 op=4 SRCH base="ou=people,dc=ms,dc=com" scope=2 filter="(&(cn=manager)(objectClass=person))" attrs="1.1"
[07/Apr/2019:04:50:15.998997938 -0400] conn=73 op=4 RESULT err=0 tag=101 nentries=1 etime=0.0000285909
[07/Apr/2019:04:50:15.999820665 -0400] conn=78 fd=74 slot=74 connection from 172.20.40.144 to 172.20.40.145
[07/Apr/2019:04:50:16.000158555 -0400] conn=78 op=0 BIND dn="cn=Directory Manager" method=128 version=3
[07/Apr/2019:04:50:16.000208944 -0400] conn=78 op=0 RESULT err=0 tag=97 nentries=0 etime=0.0000088098 dn="cn=directory manager"
[07/Apr/2019:04:50:16.000435945 -0400] conn=78 op=1 BIND dn="cn=manager,ou=People,dc=ms,dc=com" method=128 version=3
[07/Apr/2019:04:50:16.000595855 -0400] conn=78 op=1 RESULT err=0 tag=97 nentries=0 etime=0.0000186900 dn="cn=manager,ou=people,dc=ms,dc=com"
[07/Apr/2019:04:50:16.001756346 -0400] conn=73 op=5 SRCH base="ou=groups,dc=ms,dc=com" scope=2 filter="(&(memberUid=cn=manager,ou=People,dc=ms,dc=com)(objectClass=groupOfUniqueNames))" attrs="cn"
[07/Apr/2019:04:50:16.001887439 -0400] conn=73 op=5 RESULT err=0 tag=101 nentries=1 etime=0.0002866443 notes=U
[07/Apr/2019:04:50:16.002557064 -0400] conn=73 op=6 SRCH base="ou=groups,dc=ms,dc=com" scope=2 filter="(&(memberUid=cn=managers,ou=Groups,dc=ms,dc=com)(objectClass=groupOfUniqueNames))" attrs="cn"
[07/Apr/2019:04:50:16.002616354 -0400] conn=73 op=6 RESULT err=0 tag=101 nentries=0 etime=0.0000711509 notes=U
[07/Apr/2019:04:50:58.697434110 -0400] conn=71 op=10 SRCH base="ou=people,dc=ms,dc=com" scope=2 filter="(&(cn=Deployer)(objectClass=person))" attrs="1.1"
[07/Apr/2019:04:50:58.697654188 -0400] conn=71 op=10 RESULT err=0 tag=101 nentries=1 etime=0.0000309407
[07/Apr/2019:04:50:58.698351383 -0400] conn=79 fd=75 slot=75 connection from 172.20.40.144 to 172.20.40.145
[07/Apr/2019:04:50:58.699618050 -0400] conn=79 op=0 BIND dn="cn=Directory Manager" method=128 version=3
[07/Apr/2019:04:50:58.699694505 -0400] conn=79 op=0 RESULT err=0 tag=97 nentries=0 etime=0.0000128638 dn="cn=directory manager"
[07/Apr/2019:04:50:58.700144973 -0400] conn=79 op=1 BIND dn="cn=Deployer,ou=People,dc=ms,dc=com" method=128 version=3
[07/Apr/2019:04:50:58.700284861 -0400] conn=79 op=1 RESULT err=0 tag=97 nentries=0 etime=0.0000161073 dn="cn=deployer,ou=people,dc=ms,dc=com"
[07/Apr/2019:04:50:58.701065738 -0400] conn=71 op=11 SRCH base="ou=groups,dc=ms,dc=com" scope=2 filter="(&(memberUid=cn=Deployer,ou=People,dc=ms,dc=com)(objectClass=groupOfUniqueNames))" attrs="cn"
[07/Apr/2019:04:50:58.701174696 -0400] conn=71 op=11 RESULT err=0 tag=101 nentries=1 etime=0.0003450597 notes=U
[07/Apr/2019:04:50:58.701733032 -0400] conn=71 op=12 SRCH base="ou=groups,dc=ms,dc=com" scope=2 filter="(&(memberUid=cn=Deployers,ou=Groups,dc=ms,dc=com)(objectClass=groupOfUniqueNames))" attrs="cn"
[07/Apr/2019:04:50:58.701797526 -0400] conn=71 op=12 RESULT err=0 tag=101 nentries=0 etime=0.0000597109 notes=U

More information about security in WebLogic can be found here.

To add groups and users using Ansible, we can use the following:

# variables
ds_managers_group: "managers"
ds_managers_group_id: "1000"
ds_manager_group_members:
  - "manager"
ds_employees_group: "employees"
ds_employees_group_id: "1001"
ds_employees_group_members:
  - "employee"
ds_deployers_group: "Deployers"
ds_deployers_group_id: "1002"
ds_deployers_group_members:
  - "Deployer"

ds_weblogic_groups:
  - {
      cn: "{{ ds_managers_group }}",
      gidNumber: "{{ ds_managers_group_id }}",
      description: 'A group for managers'
    }
  - {
      cn: "{{ ds_employees_group }}",
      gidNumber: "{{ ds_employees_group_id }}",
      description: 'A group for employees'
    }
  - {
      cn: "{{ ds_deployers_group }}",
      gidNumber: "{{ ds_deployers_group_id }}",
      description: 'Deployers can view all resource attributes and deploy applications.'
    }

ds_weblogic_users:
  - {
      cn: 'manager',
      sn: 'manager',
      uid: 'manager',
      userPassword: 'non_hashed_password',
      description: 'A manager'
    }
  - {
      cn: 'employee',
      sn: 'employee',
      uid: 'employee',
      userPassword: 'non_hashed_password',
      description: 'An employee'
    }
  - {
      cn: 'Deployer',
      sn: 'Deployer',
      uid: 'Deployer',
      userPassword: 'non_hashed_password',
      description: 'A Deployer'
    }

# tasks
- name: remove existing groups
  ldap_entry:
    dn: "{{ item }}"
    state: absent
    params: "{{ ds_ldap_authentication }}"
  loop:
    - "cn=Accounting Managers,ou=Groups,{{ ds_suffix }}"
    - "cn=HR Managers,ou=Groups,{{ ds_suffix }}"
    - "cn=QA Managers,ou=Groups,{{ ds_suffix }}"
    - "cn=PD Managers,ou=Groups,{{ ds_suffix }}"

- name: create groups
  ldap_entry:
    dn: "cn={{ item.cn }},{{ ds_groups_dn }}"
    state: present
    objectClass:
      - posixGroup
      - top
      - groupOfUniqueNames
    attributes:
      cn: "{{ item.cn }}"
      gidNumber: "{{ item.gidNumber }}"
      description: "{{ item.description }}"
    params: "{{ ds_ldap_authentication }}"
  loop: "{{ ds_weblogic_groups }}"
    
- name: create users
  ldap_entry:
    dn: "cn={{ item.cn }},{{ ds_users_dn }}"
    state: present
    objectClass:
      - top
      - person
      - organizationalPerson
      - inetOrgPerson
    attributes:
      cn: "{{ item.cn }}"
      sn: "{{ item.sn }}"
      uid: "{{ item.uid }}"
      userPassword: "{{ item.userPassword }}"
      description: "{{ item.description }}"
    params: "{{ ds_ldap_authentication }}"
  loop: "{{ ds_weblogic_users }}"

- name: users in managers group
  ldap_attr:
    dn: "cn={{ ds_managers_group }},{{ ds_groups_dn }}"
    state: present
    name: memberUid
    values: "cn={{ item }},{{ ds_users_dn }}"
    params: "{{ ds_ldap_authentication }}"
  when: not( (item is undefined) or (item is none) or (item | trim=='') )
  loop: "{{ ds_manager_group_members }}"

- name: users in employees group
  ldap_attr:
    dn: "cn={{ ds_employees_group }},{{ ds_groups_dn }}"
    state: present
    name: memberUid
    values: "cn={{ item }},{{ ds_users_dn }}"
    params: "{{ ds_ldap_authentication }}"
  when: not( (item is undefined) or (item is none) or (item | trim=='') )
  loop: "{{ ds_employees_group_members }}"

- name: users in deployers group
  ldap_attr:
    dn: "cn={{ ds_deployers_group }},{{ ds_groups_dn }}"
    state: present
    name: memberUid
    values: "cn={{ item }},{{ ds_users_dn }}"
    params: "{{ ds_ldap_authentication }}"
  when: not( (item is undefined) or (item is none) or (item | trim=='') )
  loop: "{{ ds_deployers_group_members }}"

Linux Authentication, Authorization and Role Mapping - To use an the Directory Server for identity and authentication management on Linux systems, we can use nss-pam-ldap:

This is nss-pam-ldapd which provides a Name Service Switch (NSS, nsswitch) module that allows your LDAP server to provide user account, group, host name, alias, netgroup, and basically any other information that you would normally get from /etc flat files or NIS. It also provides a Pluggable Authentication Module (PAM) to do identity and authentication management with an LDAP server on unix systems.

This is implemented using thin NSS and PAM modules which delegate to a dedicated service (nslcd) that queries the LDAP server with persistent connections, authentication, attribute translation, etc.

The NSS module was originally a fork of nss_ldap with some structural design improvements. The most important features of nss-pam-ldapd are:

light and simple NSS and PAM libraries

avoid loading LDAP and SSL libraries in all programs

separation between NSS, PAM and LDAP code

fewer connections to the LDAP server

better debugging possibilities

better performance

To set-up nss-pam-ldap we can use the following steps:

Used variables:

nss_pam_ldap_packages:
  - nss-pam-ldapd
  - nscd

Install the nss-pam-ldap packages.
Enable and start the nslcd service.

Configure nss-pam-ldap.

# This is the configuration file for the LDAP nameservice
# switch library's nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.

# The user and group nslcd should run as.
uid nslcd
gid ldap

# The uri pointing to the LDAP server to use for name lookups.
# Multiple entries may be specified. The address that is used
# here should be resolvable without using LDAP (obviously).
#uri ldap://127.0.0.1/
#uri ldaps://127.0.0.1/
#uri ldapi://%2fvar%2frun%2fldapi_sock/
# Note: %2f encodes the '/' used as directory separator
uri ldap://{{ db1_fqdn }}:{{ ds_directory_server_network_port }}

# The LDAP version to use (defaults to 3
# if supported by client library)
#ldap_version 3

# The distinguished name of the search base.
base {{ ds_suffix }}

# The distinguished name to bind to the server with.
# Optional: default is to bind anonymously.
binddn {{ ds_directory_manager_dn }}

# The credentials to bind with.
# Optional: default is no credentials.
# Note that if you set a bindpw you should check the permissions of this file.
bindpw {{ ds_directory_manager_password }}

# The distinguished name to perform password modifications by root by.
#rootpwmoddn cn=admin,dc=example,dc=com

# The default search scope.
#scope sub
#scope one
#scope base

# Customize certain database lookups.
base   group  {{ ds_groups_dn }}
base   passwd {{ ds_users_dn }}
base   shadow {{ ds_users_dn }}
#scope  group  onelevel
#scope  hosts  sub

# Bind/connect timelimit.
bind_timelimit 30

# Search timelimit.
timelimit 30

# Idle timelimit. nslcd will close connections if the
# server has not been contacted for the number of seconds.
#idle_timelimit 3600

# Use StartTLS without verifying the server certificate.
ssl no
...

Restart nslcd service.
Configure user information service in /etc/nsswitch.conf and authentication functions in /etc/pam.d/system-auth and /etc/pam.d/password-auth by using the authentication configuration tool authconfig. For more information see the System-Level Authentication Guide.
Enable custom home directories.

To combine the above in an Ansible playbook, we have the following:

- name: install nss-pam-ldap (nslcd)
  yum:
    name: "{{ nss_pam_ldap_packages }}"
    state: present
    
- name: enable and start nslcd service
  systemd:
    name: nslcd
    state: started
    enabled: yes
  register: nslcd_started

- name: configure nss-pam-ldap (nslcd)
  template:
    src: roles/demo_nss_pam_ldap/templates/nslcd_conf.j2
    dest: "{{ nss_pam_ldap_conf_path }}"
    owner: root
    group: root
    mode: 0600
  register: nslcd_configured

- name: restart nslcd service
  systemd:
    name: nslcd
    state: restarted
  when: nslcd_configured is changed

- name: check if authconfig ldap is enabled
  lineinfile:
    path: /etc/nsswitch.conf
    line: "passwd:     files sss ldap"
    state: present
  check_mode: yes
  register: authconfig_ldap_enabled

- name: debug authconfig
  debug:
    var=authconfig_ldap_enabled
  when: authconfig_ldap_enabled is changed

- name: execute authconfig enable ldap
  command: /bin/sh -c "/sbin/authconfig --update --enableldap --enableldapauth"
  when: authconfig_ldap_enabled is changed

- name: check if authconfig mkhomedir is enabled
  lineinfile:
    path: /etc/pam.d/system-auth-ac
    line: "session     optional      pam_mkhomedir.so umask=0077"
    state: present
  check_mode: yes
  register: authconfig_mkhomedir_enabled

- name: debug authconfig
  debug:
    var=authconfig_mkhomedir_enabled
  when: authconfig_mkhomedir_enabled is changed

- name: execute authconfig enable mkhomedir
  command: /bin/sh -c "/sbin/authconfig --update --enablemkhomedir"
  when: authconfig_mkhomedir_enabled is changed

To test the set-up, we add a (Linux) user and group to the Directory Server

# variables
ds_operator_group: "operators"
ds_operator_group_id: "5000"
    
ds_linux_groups:
  - {
      cn: "{{ ds_operator_group }}",
      gidNumber: "{{ ds_operator_group_id }}",
      description: 'Group for operators'
    }
    
ds_linux_users:
  - {
      cn: 'FirstName',
      sn: 'LastName',
      uid: 'testuser',
      userPassword: 'non_hashed_password',
      uidNumber: '10000',
      gidNumber: "{{ ds_operator_group_id }}",
      homeDirectory: '/home/testuser',
      loginShell: '/bin/bash',
      gecos: 'FullName',
      sshPublicKey: 'ssh-rsa AAAAB3NzaC1yc2EA ... BA5X+SZ77dUUDWKv'
    }

# tasks
- name: create linux groups
  ldap_entry:
    dn: "cn={{ item.cn }},{{ ds_groups_dn }}"
    state: present
    objectClass:
      - posixGroup
    attributes:
      cn: "{{ item.cn }}"
      gidNumber: "{{ item.gidNumber }}"
      description: "{{ item.description }}"
    params: "{{ ds_ldap_authentication }}"
  loop: "{{ ds_linux_groups }}"
    
- name: create linuxusers
  ldap_entry:
    dn: "uid={{ item.uid }},{{ ds_users_dn }}"
    state: present
    objectClass:
      - ldapPublicKey
      - posixAccount
      - shadowAccount
      - inetOrgPerson
    attributes:
      cn: "{{ item.cn }}"
      sn: "{{ item.sn }}"
      uid: "{{ item.uid }}"
      userPassword: "{{ item.userPassword }}"
      uidNumber: "{{ item.uidNumber }}"
      gidNumber: "{{ item.gidNumber }}"
      homeDirectory: "{{ item.homeDirectory }}"
      loginShell: "{{ item.loginShell }}"
      gecos: "{{ item.gecos }}"
      sshPublicKey: "{{ item.sshPublicKey }}"
    params: "{{ ds_ldap_authentication }}"
  loop: "{{ ds_linux_users }}"

When the user and group is not present we have the following:

[root@db1 /]# ls -l home/
total 0
drwx------. 1 ansible ansible  146 Mar 15 08:41 ansible
drwx------. 1 oracle  oinstall  92 Mar 16 06:34 oracle

[root@db1 /]# id testuser
id: testuser: no such user

When we add the user and group to the Directory Server we have the following

# login using ssh key
[mobaxterm] ➤ ssh -i id_rsa testuser@db1.ms.com
...
Creating directory '/home/testuser'.
...

[testuser@db1 ~]$ ls -altr
total 16
-rw-------  1 testuser operators 172 Apr  7 07:52 .kshrc
-rw-------  1 testuser operators 231 Apr  7 07:52 .bashrc
-rw-------  1 testuser operators 193 Apr  7 07:52 .bash_profile
-rw-------  1 testuser operators  18 Apr  7 07:52 .bash_logout
drwxr-xr-x. 1 root     root       42 Apr  7 07:52 ..
drwx------  1 testuser operators  76 Apr  7 07:52 .

[testuser@db1 ~]$ id testuser
uid=10000(testuser) gid=5000(operators) groups=5000(operators)

[testuser@db1 ~]$ yum install dos2unix
Loaded plugins: ulninfo
You need to be root to perform this command.

[testuser@db1 ~]$ sudo yum install dos2unix
[sudo] password for testuser:
Loaded plugins: ulninfo
Package dos2unix-6.0.3-7.el7.x86_64 already installed and latest version
Nothing to do

An alternative to nss-pam-ldap (nscd) is sssd. As sssd is not designed to be used with nscd (more information can be found here), we will configure sssd on the test host. To configure sssd the following steps can be used:

Used variables:

sssd_packages:
  - sssd
  - sssd-tools

sssd_conf_path: "/etc/sssd/sssd.conf"

Install the sssd packages.

Configure sssd to use the Directory Server, more information can be found here, for example,

# Config documentation links
#
# https://linux.die.net/man/5/sssd.conf
# https://linux.die.net/man/5/sssd-ldap
# https://linux.die.net/man/8/sss_cache
# https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/system-level_authentication_guide/index#SSSD
#
#
[sssd]
config_file_version = 2
services = nss, pam
domains = test151

[nss]
filter_groups = root
filter_users = root
entry_cache_nowait_percentage = 75

[pam]
offline_credentials_expiration = 2
offline_failed_login_attempts = 3
offline_failed_login_delay = 5

[domain/test151]
ldap_uri = {{ ds_ldaps_authentication.server_uri }}
ldap_search_base = {{ ds_suffix }}
ldap_schema = rfc2307

id_provider = ldap
ldap_id_use_start_tls = true
enumerate = false
cache_credentials = true
entry_cache_timeout = 300

auth_provider = ldap
#ldap_tls_cacertdir = {{ ds_nss_database_path }}
ldap_tls_cacert = {{ ds_nss_database_ca_certificate_path }}
ldap_tls_reqcert = allow
ldap_default_bind_dn = {{ ds_ldaps_authentication.bind_dn }}
ldap_default_authtok_type = password
ldap_default_authtok = {{ ds_ldaps_authentication.bind_pw }}
ldap_group_search_base = {{ ds_groups_dn }}
ldap_user_search_base = {{ ds_users_dn }}

chpass_provider = ldap

access_provider = ldap
ldap_access_filter = (&(objectclass=posixAccount)(gidNumber=5000))

above also the system services nss and pam are configured (more information can be found here).

Configure user information service in /etc/nsswitch.conf and authentication functions in /etc/pam.d/system-auth and /etc/pam.d/password-auth by using the authentication configuration tool authconfig. For more information see the System-Level Authentication Guide.
Enable custom home directories.

To combine the above in an Ansible playbook, we have the following:

- name: install sssd
  yum:
    name: "{{ sssd_packages }}"
    state: present
    
- name: enable and start sssd service
  systemd:
    name: sssd
    state: started
    enabled: yes
  register: sssd_started

- name: configure sssd
  template:
    src: roles/demo_sssd/templates/sssd_conf.j2
    dest: "{{ sssd_conf_path }}"
    owner: root
    group: root
    mode: 0600
  register: sssd_configured

- name: restart sssd service
  systemd:
    name: sssd
    state: restarted
  when: sssd_configured is changed

- name: check if authconfig sssd is enabled
  lineinfile:
    path: /etc/pam.d/system-auth
    line: "auth        [default=1 success=ok] pam_localuser.so"
    state: present
  check_mode: yes
  register: authconfig_sssd_enabled

- name: debug authconfig
  debug:
    var=authconfig_sssd_enabled
  when: authconfig_sssd_enabled is changed

- name: execute authconfig enable sssd
  command: /bin/sh -c "/sbin/authconfig --update --enablesssd --enablesssdauth"
  when: authconfig_sssd_enabled is changed

- name: check if authconfig mkhomedir is enabled
  lineinfile:
    path: /etc/pam.d/system-auth-ac
    line: "session     optional      pam_mkhomedir.so umask=0077"
    state: present
  check_mode: yes
  register: authconfig_mkhomedir_enabled

- name: debug authconfig
  debug:
    var=authconfig_mkhomedir_enabled
  when: authconfig_mkhomedir_enabled is changed

- name: execute authconfig enable mkhomedir
  command: /bin/sh -c "/sbin/authconfig --update --enablemkhomedir"
  when: authconfig_mkhomedir_enabled is changed

With the configuration in place let us test the set-up

[root@test151 ~]# systemctl start sssd

[root@test151 ~]# ll /var/lib/sss/db

[root@test151 ~]# tail -f /var/log/dirsrv/slapd-test151/access

[root@test151 ~]# id testuser
uid=10000(testuser) gid=5000(operators) groups=5000(operators)

[root@test151 ~]# getent passwd testuser
testuser:*:10000:5000:FullName:/home/testuser:/bin/bash

[root@test151 ~]# sssctl user-checks testuser --action=auth
user: testuser
action: auth
service: system-auth

SSSD nss user lookup result:
 - user name: testuser
 - user id: 10000
 - group id: 5000
 - gecos: FullName
 - home directory: /home/testuser
 - shell: /bin/bash

SSSD InfoPipe user lookup result:
 - name: testuser
 - uidNumber: 10000
 - gidNumber: 5000
 - gecos: FullName
 - homeDirectory: /home/testuser
 - loginShell: /bin/bash

testing pam_authenticate

Password:
pam_authenticate for user [testuser]: Success

PAM Environment:
 - no env -
 
[root@test151 ~]# systemctl stop sssd

[root@test151 ~]# id testuser
uid=10000(testuser) gid=5000(operators) groups=5000(operators)

[root@test151 ~]# sss_cache -E

[root@test151 ~]# id testuser
id: testuser: no such user

[root@test151 ~]# systemctl start sssd

[root@test151 ~]# sssctl user-checks testuser
user: testuser
action: acct
service: system-auth

SSSD nss user lookup result:
 - user name: testuser
 - user id: 10000
 - group id: 5000
 - gecos: FullName
 - home directory: /home/testuser
 - shell: /bin/bash

SSSD InfoPipe user lookup result:
 - name: testuser
 - uidNumber: 10000
 - gidNumber: 5000
 - gecos: FullName
 - homeDirectory: /home/testuser
 - loginShell: /bin/bash

testing pam_acct_mgmt

pam_acct_mgmt: Success

PAM Environment:
 - no env -
 
[root@test151 ~]# sssctl domain-status test151
Online status: Online
Active servers:
LDAP: test151.ms.com
Discovered LDAP servers:
- test151.ms.com

[root@test151 ~]# sssctl user-show testuser
Name: testuser
Cache entry creation date: 04/19/19 09:33:31
Cache entry last update time: 04/19/19 09:46:22
Cache entry expiration time: Expired
Initgroups expiration time: Expired
Cached in InfoPipe: No

When things are not working troubleshooting tips are given here.

Application & Infrastructure Monitoring Authentication, Authorization and Role Mapping - For the application and infrastructure monitoring we use Check_MK. To set-up Check_MK we can use the following steps:

Used variables:

cmk_software_directory: "{{ software_directory }}/check_mk"
cmk_install_file: "check-mk-enterprise-1.5.0p14-el7-38.x86_64.rpm"

cmk_site_name: "support"
cmk_site_conf_path: "/opt/omd/sites/{{ cmk_site_name }}/etc/omd/site.conf"

cmk_listen_port: "80"

Add the optional packages yum repository, for more information see the documentation here
In order to be able to use EPEL on RedHat, the package sources for optional RPMs (at least for RedHat 6) are required, if these haven't already been installed during the operating system installation. Without these sources the freeradius-utils, graphviz-gd, and php-mbstring package will be missing.
Copy the Check_MK install file (rpm).
Create a monitoring instance, by using /usr/bin/omd create {{ cmk_site_name }}.
Start the created monitoring instance.
Optionally change the admin passwords by using the htpasswd module.

To combine the above in an Ansible playbook, we have the following:

# https://yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64/index.html
- name: add oracle linux 7 (x86_64) optional latest
  yum_repository:
    name: epel_optional_packages
    description: Oracle Linux 7 (x86_64) Optional Latest
    baseurl: http://yum.oracle.com/repo/OracleLinux/OL7/optional/latest/x86_64
    gpgcheck: no
    enabled: yes

- name: create software deploy directory
  file:
    path: "{{ cmk_software_directory }}"
    state: directory

- name: copy latest {{ cmk_install_file }}
  copy:
    src: "/home/ansible/{{ cmk_install_file }}"
    dest: "{{ cmk_software_directory }}"
  register: check_mk_software_copied

- name: install check_mk
  yum:
    name: "{{ cmk_software_directory }}/{{ cmk_install_file }}"
    state: present
  when: check_mk_software_copied is success
  register: check_mk_installed

- name: create monitoring instance {{ cmk_site_name }}
  command: /bin/sh -c "/usr/bin/omd create {{ cmk_site_name }}"
    creates="{{ cmk_site_conf_path }}"
  when: check_mk_installed is success
  register: check_mk_instance_created
  
- name: debug monitoring instance {{ cmk_site_name }}
  debug:
    var=check_mk_instance_created
  when: check_mk_instance_created is changed

- name: start monitoring instance {{ cmk_site_name }}
  systemd:
    name: omd
    state: started
    enabled: yes
  register: omd_started
  
- name: change admin passwords
  htpasswd: 
    path: "/omd/sites/{{ cmk_site_name }}/etc/htpasswd"
    crypt_scheme: apr_md5_crypt
    name: "{{ item }}"
    password: password
    owner: "{{ cmk_site_name }}"
    group: "{{ cmk_site_name }}"
    mode: 0660
  loop:
    - cmkadmin
    - automation

With Check_MK installed and configured, we add an LDAP connection:

The tool for configuring Check_MK via GUI is called the WATO (Web Administration Tool). Access the user interface (using the cmkadmin user), and click WATO-Configuration, Users, LDAP Connections, and add the following:

--- GENERAL PROPERTIES
- ID: test
- Description: test users
--- LDAP CONNECTION
- Directory Type: 389 Directory Server
    - Connect to
        - Manually specify list of LDAP servers
            - LDAP Server: db1.ms.com
- Bind Credentials
    - Bind DN: cn=Directory Manager
    - Bind Password: password
- TCP Port: 389
--- USERS
- User Base DN: ou=people,dc=ms,dc=com
- Search Scope: Search all entries one level below the base DN
- Search Filter: (&(objectclass=posixAccount)(gidNumber=5000))
- User-ID Attribute: uid
--- GROUPS
- Group Base DN: ou=groups,dc=ms,dc=com
- Search Scope: Search all entries one level below the base DN
- Search Filter: (objectclass=groupOfUniqueNames)
- Member Attribute: memberUid
--- ATTRIBUTE SYNC PLUGINS 
- Alias, LDAP attribute to sync: cn
- Authentication Expiration, LDAP attribute to be used as indicator: shadowExpire
- Contactgroup Membership
- Roles, Administrator, Group DN: cn=employees,ou=groups,dc=ms,dc=com, Search In: This connection
--- OTHER
- Sync interval: 1 days 0 hours 0 mins

With the configuration in place click save and test and make the changes permanent by activatating the changes. Next, synchronize the users, and login with the testuser account.

I spent my entire life getting ready for this sort of future and it's going down easy - Wally.

References

[1] Ansible Documentation.

Message-Driven AQ

2017-12-10T12:07:00.001+01:00

In this post we are going to combine message-driven beans ( running on WebLogic ) with Oracle Database Advanced Queuing.

To configure AQ is described in the document Interoperating with Oracle AQ JMS:

connect / as sysdba;

-- create a new user
create user jmsuser identified by jmsuserpwd;
alter user jmsuser default tablespace users;
alter user jmsuser temporary tablespace temp;
grant dba to jmsuser;
grant aq_user_role to jmsuser;
grant execute on sys.dbms_aqadm to jmsuser;
grant execute on sys.dbms_aq to jmsuser;
grant execute on sys.dbms_aqin to jmsuser;
grant execute on sys.dbms_aqjms to jmsuser;

connect jmsuser / jmsuserpwd;

-- create AQ tables dbms_aqadm.create_queue_table
execute dbms_aqadm.create_queue_table(queue_table=>'TestQueueTable', queue_payload_type=>'sys.aq$_jms_text_message', multiple_consumers=>false);

-- create queue dbms_aqadm.create_queue
execute dbms_aqadm.create_queue(queue_name=>'TestUserQueue', queue_table=>'TestQueueTable', max_retries=>1, retry_delay=>120);

-- start the queue dbms_aqadm.start_queue
execute dbms_aqadm.start_queue(queue_name=>'TestUserQueue');

As we will be working with message-driven beans (acknowledgments and container managed transations), the data source and foreign server connection factory must be XA-enabled. The data source is configured as follows

<jdbc-data-source ...>
    <name>aq_test_ds</name>
    <datasource-type>GENERIC</datasource-type>
    <jdbc-driver-params>
        <url>jdbc:oracle:thin:@//datanode.ms.com:1521/orcl12</url>
        <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name>
        <properties>
            <property>
                <name>user</name>
                <value>jmsuser</value>
            </property>
        </properties>
        <password-encrypted>{AES}LjD9zLInpLXURZ12kqQdcGIHqcjlIaTqRzu2U5WkJR0=</password-encrypted>
    </jdbc-driver-params>
    <jdbc-connection-pool-params>
        <test-table-name>SQL ISVALID</test-table-name>
    </jdbc-connection-pool-params>
    <jdbc-data-source-params>
        <jndi-name>jdbc/aq_test_ds</jndi-name>
        <global-transactions-protocol>TwoPhaseCommit</global-transactions-protocol>
    </jdbc-data-source-params>
</jdbc-data-source>

WebLogic AQ JMS supports both Global XA (JTA) transactions and local JMS transacted session transactions. Global transactions require use of XA based connection factories, while local transactions use non-XA based JMS connection factories:

When selecting a non-XA JDBC driver, then we can only use WebLogic AQ JMS in local transactions.
When selecting an XA JDBC driver, then we can use WebLogic AQ JMS in both local and global transactions.
Note that this release does not support non-XA JDBC driver data sources with any of the global transaction options such as Logging Last Resource (LLR), One-Phase Commit (JTS), and Emulated Two-Phase Commit. If Supports Global Transactions is selected, then WebLogic Server logs a warning message.
Global transactions are supported only with an XA JDBC driver One-Phase commit optimization. If we use the same XA capable data source for both AQ JMS and JDBC operations, then the XA transactional behavior is equivalent to having two connections in a single data source that is treated as a single resource by the transaction manager. Therefore, if the AQ JMS and JDBC operations are called under the same JTA transaction, and no other resources are involved in the transaction, the transaction uses One-Phase Commit optimization instead of Two-Phase Commit; otherwise read-only optimization is used. More information can be found in the document Understanding Transactions.

The foreign server is configured as follows

<weblogic-jms ...>
    <foreign-server name="aq_test_foreign_server">
        <default-targeting-enabled>true</default-targeting-enabled>
        <foreign-destination name="aq_test_foreign_destination">
            <local-jndi-name>jms/AQTestQueue</local-jndi-name>
            <remote-jndi-name>Queues/TestUserQueue</remote-jndi-name>
        </foreign-destination>
        <foreign-connection-factory name="aq_test_foreign_connection_factory">
            <local-jndi-name>jms/AQTestConnectionFactory</local-jndi-name>
            <remote-jndi-name>XAQueueConnectionFactory</remote-jndi-name>
            <connection-health-checking>enabled</connection-health-checking>
        </foreign-connection-factory>
        <initial-context-factory>oracle.jms.AQjmsInitialContextFactory</initial-context-factory>
        <jndi-property>
            <key>datasource</key>
            <value>jdbc/aq_test_ds</value>
        </jndi-property>
    </foreign-server>
</weblogic-jms>

With the resources in-place, we continue with creating the message-driven bean. Message-driven beans, like other well-behaved MessageListeners, should not, in general, throw runtime exceptions. If a message-driven bean's onMessage method encounters a system-level exception or error that does not allow the method to successfully complete, the Enterprise JavaBeans Specification (JSR-000345 EJB 3.2 Core Specification, 9.2.2 System Exceptions) provides the following guidelines:

If the enterprise bean method encounters a system-level exception or error that does not allow the method to successfully complete, the method should throw a suitable non-application exception that is compatible with the method’s throws clause. While the EJB specification does not prescribe the exact usage of the exception, it encourages the Bean Provider to follow these guidelines:
- If the bean method encounters a system exception or error, it should simply propagate the error from the bean method to the container (i.e., the bean method does not have to catch the exception).
- If the bean method performs an operation that results in a checked exception that the bean method cannot recover, the bean method should throw the javax.ejb.EJBException that wraps the original exception.
- Any other unexpected error conditions should be reported using the javax.ejb.EJBException.

Note that the javax.ejb.EJBException is a subclass of the java.lang.RuntimeException, and therefore it does not have to be listed in the throws clauses of the business methods.

Under container-managed transaction demarcation, upon receiving a runtime exception from a message-driven bean's onMessage method, the container rolls back the container-started transaction and the message is redelivered (as was configured in the creation of de AQ queue - max_retries=1 and retry_delay=120). This is because the message delivery itself is part of the container-started transaction. By default, the Application Server container closes the container's connection to the JMS provider when the first runtime exception is received from a message-driven bean instance's onMessage method. This avoids potential message redelivery looping and protects server resources if the message-driven bean's onMessage method continues misbehaving.

Until a JMS message has been acknowledged, it is not considered to be successfully consumed. The successful consumption of a message ordinarily takes place in three stages:

The client receives the message.
The client processes the message.
The message is acknowledged. Acknowledgment is initiated either by the JMS provider or by the client, depending on the session acknowledgment mode.

In transacted sessions, acknowledgment happens automatically when a transaction is committed. If a transaction is rolled back, all consumed messages are redelivered. In nontransacted sessions, when and how a message is acknowledged depend on the value specified as the second argument of the createSession method. The three possible argument values are as follows:

Session.AUTO_ACKNOWLEDGE: The session automatically acknowledges a client's receipt of a message either when the client has successfully returned from a call to receive or when the MessageListener it has called to process the message returns successfully. A synchronous receive in an AUTO_ACKNOWLEDGE session is the one exception to the rule that message consumption is a three-stage process as described earlier. In this case, the receipt and acknowledgment take place in one step, followed by the processing of the message.
Session.CLIENT_ACKNOWLEDGE: A client acknowledges a message by calling the message's acknowledge method. In this mode, acknowledgment takes place on the session level: Acknowledging a consumed message automatically acknowledges the receipt of all messages that have been consumed by its session. For example, if a message consumer consumes ten messages and then acknowledges the fifth message delivered, all ten messages are acknowledged. In the Java EE platform, a CLIENT_ACKNOWLEDGE session can be used only in an application client, not in a web component or enterprise bean.
Session.DUPS_OK_ACKNOWLEDGE: This option instructs the session to lazily acknowledge the delivery of messages. This is likely to result in the delivery of some duplicate messages if the JMS provider fails, so it should be used only by consumers that can tolerate duplicate messages. (If the JMS provider redelivers a message, it must set the value of the JMSRedelivered message header to true.) This option can reduce session overhead by minimizing the work the session does to prevent duplicates.

If messages have been received from a queue but not acknowledged when a session terminates, the JMS provider retains them and redelivers them when a consumer next accesses the queue. The provider also retains unacknowledged messages for a terminated session that has a durable TopicSubscriber. Unacknowledged messages for a nondurable TopicSubscriber are dropped when the session is closed.

When using a queue or a durable subscription, we can use the Session.recover method to stop a nontransacted session and restart it with its first unacknowledged message. In effect, the session's series of delivered messages is reset to the point after its last acknowledged message. The messages it now delivers may be different from those that were originally delivered, if messages have expired or if higher-priority messages have arrived. For a nondurable TopicSubscriber, the provider may drop unacknowledged messages when its session is recovered. More information can be found in the document Java Message Service Concepts. An example message-driven bean is given in the document The Message-Driven Bean Class. Our message-driven bean class looks as follows

package model.logic;

import javax.ejb.EJBException;
import javax.jms.*;

public class AQTestMDB implements MessageListener {
    @Override
    public void onMessage(Message message) {
        if (message instanceof TextMessage) {
            TextMessage textMessage = (TextMessage) message;

            String text = null;

            try {
                text = textMessage.getText();
            } catch (JMSException e) {
                e.printStackTrace();
            }

            if (text.contains("exception")) {
                throw new EJBException("text contains exception");
            } else {
                System.out.println("RECEIVED TEXT MESSAGE " + text);
            }
        }
    }
}

The document Settings for Message Driven Beans to Interoperate with AQ JMS contains information on how the message-driven bean should be configured. In our case, we use the following:

META-INF/ejb-jar.xml
<ejb-jar ...>
    <enterprise-beans>
        <message-driven>
            <ejb-name>AQTestMDB</ejb-name>
            <mapped-name>jms/AQTestQueue</mapped-name>
            <ejb-class>model.logic.AQTestMDB</ejb-class>
            <transaction-type>Container</transaction-type>
            <message-destination-type>javax.jms.Queue</message-destination-type>
        </message-driven>
    </enterprise-beans>
</ejb-jar>

META-INF/weblogic-ejb-jar.xml
<weblogic-ejb-jar ...>
    <weblogic-enterprise-bean>
        <ejb-name>AQTestMDB</ejb-name>
        <message-driven-descriptor>
            <pool>
                <max-beans-in-free-pool>1</max-beans-in-free-pool>
                <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
            </pool>
            <destination-jndi-name>jms/AQTestQueue</destination-jndi-name>
            <connection-factory-jndi-name>jms/AQTestConnectionFactory</connection-factory-jndi-name>
        </message-driven-descriptor>
    </weblogic-enterprise-bean>
</weblogic-ejb-jar>

Package the application and deploy it to WebLogic.

To test the set-up we need to put a few messages on the queue. The document More Oracle Database Advanced Queuing JMS Examples shows how do accomplish this, i.e.,

create or replace procedure enqueue_text( text in varchar2 ) as
    agent               sys.aq$_agent := sys.aq$_agent(' ', null, 0);
    message             sys.aq$_jms_text_message;
    enqueue_options     dbms_aq.enqueue_options_t;
    message_properties  dbms_aq.message_properties_t;
    msgid               raw(16);
begin
    message := sys.aq$_jms_text_message.construct;
    message.set_text(text);
    
    dbms_aq.enqueue(queue_name => 'testuserqueue', enqueue_options => enqueue_options, message_properties => message_properties, payload => message, msgid => msgid);
end;

which can be called as follows

declare
    text varchar2(200);
begin
    text := 'A little bit of good text';
    enqueue_text( text => text );
end;

In the case the variable text contains the word exception an exception will be raised

<Nov 29, 2017 2:00:58,485 PM CET> <Warning> <EJB> <BEA-010065> <MessageDrivenBean threw an Exception in onMessage(). The exception is:
 javax.ejb.EJBException: text contains exception.
javax.ejb.EJBException: text contains exception
        at model.logic.AQTestMDB.onMessage(AQTestMDB.java:30)
        at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:438)
        at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:361)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.processOneMessage(TokenBasedJMSMessagePoller.java:355)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.run(TokenBasedJMSMessagePoller.java:125)
        Truncated. see log file for complete stacktrace
>
<Nov 29, 2017 2:00:58,507 PM CET> <Error> <EJB> <BEA-010080> <An error occurred while attempting to process a message inside a Message-Driven Bean: javax.ejb.EJBException: text contains exception
Exception is : javax.ejb.EJBException: text contains exception
        at model.logic.AQTestMDB.onMessage(AQTestMDB.java:30)
        at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:438)
        at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:361)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.processOneMessage(TokenBasedJMSMessagePoller.java:355)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.run(TokenBasedJMSMessagePoller.java:125)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
>
<Nov 29, 2017 2:02:59,588 PM CET> <Error> <EJB> <BEA-010080> <An error occurred while attempting to process a message inside a Message-Driven Bean: javax.ejb.EJBException: text contains exception
Exception is : javax.ejb.EJBException: text contains exception
        at model.logic.AQTestMDB.onMessage(AQTestMDB.java:30)
        at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:438)
        at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:361)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.processOneMessage(TokenBasedJMSMessagePoller.java:355)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.run(TokenBasedJMSMessagePoller.java:125)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
>
RECEIVED TEXT MESSAGE A little bit of good text

When the message does not contain the word exception the text RECEIVED TEXT MESSAGE ... will be printed in the WebLogic .out-file.

As can be seen from the logging, a 'wrong' message is redelivered after two minutes (after which it again fails and is redirected to an error-queue).

In WebLogic we have the following statistics for the message-driven bean

Two commited transactions (text does not contain the word exception), two transactions rolled back (one message containing the word exception and a redilvery of that message).

To make things a bit more interesting; instead of writing the message to a log file, we produce the message to a WebLogic Queue. To accomplish this we change the code to

package model.logic;

import javax.ejb.EJBException;
import javax.jms.*;

public class AQTestMDB implements MessageListener {

    private ConnectionFactory testConnectionFactory;
    private Queue testQueue;

    @Override
    public void onMessage(Message message) {
        if (message instanceof TextMessage) {
            TextMessage textMessage = (TextMessage) message;

            String text = null;

            try {
                text = textMessage.getText();
            } catch (JMSException e) {
                throw new EJBException(e);
            }

            if (text.contains("exception")) {
                throw new EJBException("text contains exception");
            } else {
                try {
                    sendMessage(text);
                    System.out.println("TEXT MESSAGE " + text + " SEND.");
                } catch (JMSException e) {
                    throw new EJBException(e);
                }
            }
        }
    }

    private void sendMessage(String text) throws JMSException {
        Connection connection = null;
        Session session = null;
        try {
            connection = testConnectionFactory.createConnection();
            session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
            MessageProducer messageProducer = session.createProducer(testQueue);

            TextMessage message = session.createTextMessage();
            message.setText(text);
            messageProducer.send(message);
        } finally {
            if (session != null) {
                session.close();
            }
            if (connection != null) {
                connection.close();
            }
        }
    }
}

with the following configuration

ejb-jar.xml
<ejb-jar ...>
    <enterprise-beans>
        <message-driven>
            <ejb-name>AQTestMDB</ejb-name>
            <mapped-name>jms/AQTestQueue</mapped-name>
            <ejb-class>model.logic.AQTestMDB</ejb-class>
            <messaging-type>javax.jms.MessageListener</messaging-type>
            <transaction-type>Container</transaction-type>
            <message-destination-type>javax.jms.Queue</message-destination-type>

            <resource-ref>
                <res-ref-name>jms/TestConnectionFactory</res-ref-name>
                <res-type>javax.jms.ConnectionFactory</res-type>
                <res-auth>Container</res-auth>
                <res-sharing-scope>Shareable</res-sharing-scope>
                <injection-target>
                    <injection-target-class>model.logic.AQTestMDB</injection-target-class>
                    <injection-target-name>testConnectionFactory</injection-target-name>
                </injection-target>
            </resource-ref>

            <resource-ref>
                <res-ref-name>jms/TestQueue</res-ref-name>
                <res-type>javax.jms.Queue</res-type>
                <res-auth>Container</res-auth>
                <res-sharing-scope>Shareable</res-sharing-scope>
                <injection-target>
                    <injection-target-class>model.logic.AQTestMDB</injection-target-class>
                    <injection-target-name>testQueue</injection-target-name>
                </injection-target>
            </resource-ref>

        </message-driven>
    </enterprise-beans>
</ejb-jar>

 
weblogic-ejb-jar.xml (not changed)
<weblogic-ejb-jar ...>
    <weblogic-enterprise-bean>
        <ejb-name>AQTestMDB</ejb-name>
        <message-driven-descriptor>
            <pool>
                <max-beans-in-free-pool>1</max-beans-in-free-pool>
                <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
            </pool>
            <destination-jndi-name>jms/AQTestQueue</destination-jndi-name>
            <connection-factory-jndi-name>jms/AQTestConnectionFactory</connection-factory-jndi-name>
        </message-driven-descriptor>
    </weblogic-enterprise-bean>
</weblogic-ejb-jar>

In WebLogic, we have added the following connection factory and distributed queue

<weblogic-jms ...>
    <connection-factory name="TestConnectionFactory">
        <default-targeting-enabled>true</default-targeting-enabled>
        <jndi-name>jms/TestConnectionFactory</jndi-name>
        <client-params>
            <client-id-policy>Restricted</client-id-policy>
            <subscription-sharing-policy>Exclusive</subscription-sharing-policy>
            <messages-maximum>10</messages-maximum>
        </client-params>
        <transaction-params>
            <xa-connection-factory-enabled>true</xa-connection-factory-enabled>
        </transaction-params>
        <security-params>
            <attach-jmsx-user-id>false</attach-jmsx-user-id>
        </security-params>
    </connection-factory>
    <foreign-server name="aq_test_foreign_server">
        ...
    </foreign-server>
    <uniform-distributed-queue name="TestQueue">
        <sub-deployment-name>aq_test_sub_deployment</sub-deployment-name>
        <default-targeting-enabled>false</default-targeting-enabled>
        <jndi-name>jms/TestQueue</jndi-name>
        <load-balancing-policy>Round-Robin</load-balancing-policy>
        <forward-delay>-1</forward-delay>
        <reset-delivery-count-on-forward>true</reset-delivery-count-on-forward>
    </uniform-distributed-queue>
</weblogic-jms>

To test the set-up, we put some messages on the AQ queue

declare
    text varchar2(200);
begin
    text := 'something exceptional';
    enqueue_text( text => text );
end;

After sending a pair of messages the WebLogic logging shows the following

TEXT MESSAGE something useful SEND.
<Dec 5, 2017 10:41:34,609 AM CET> <Warning> <EJB> <BEA-010065> <MessageDrivenBean threw an Exception in onMessage(). The exception is:
 javax.ejb.EJBException: text contains exception.
javax.ejb.EJBException: text contains exception
        at model.logic.AQTestMDB.onMessage(AQTestMDB.java:25)
        at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:438)
        at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:361)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.processOneMessage(TokenBasedJMSMessagePoller.java:355)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.run(TokenBasedJMSMessagePoller.java:125)
        Truncated. see log file for complete stacktrace
>
<Dec 5, 2017 10:41:34,624 AM CET> <Error> <EJB> <BEA-010080> <An error occurred while attempting to process a message inside a Message-Driven Bean: javax.ejb.EJBException: text contains exception
Exception is : javax.ejb.EJBException: text contains exception
        at model.logic.AQTestMDB.onMessage(AQTestMDB.java:25)
        at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:438)
        at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:361)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.processOneMessage(TokenBasedJMSMessagePoller.java:355)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.run(TokenBasedJMSMessagePoller.java:125)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
>
<Dec 5, 2017 10:43:35,661 AM CET> <Error> <EJB> <BEA-010080> <An error occurred while attempting to process a message inside a Message-Driven Bean: javax.ejb.EJBException: text contains exception
Exception is : javax.ejb.EJBException: text contains exception
        at model.logic.AQTestMDB.onMessage(AQTestMDB.java:25)
        at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:438)
        at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:361)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.processOneMessage(TokenBasedJMSMessagePoller.java:355)
        at weblogic.ejb.container.internal.TokenBasedJMSMessagePoller.run(TokenBasedJMSMessagePoller.java:125)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:670)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:644)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:415)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:355)
>

Again a 'wrong' message is retried, and after a retry put in an AQ error queue

while a 'good' message is put in the WebLogic queue

for further processing.

Information to tune message-driven beans can be found in the document Tuning Message-Driven Beans. Of special interrest is the message-driven bean thread management

Number of concurrent message-driven beans: min(max-beans-in-free-pool,16)
Transactional WebLogic message-driven beans use a synchronous polling mechanism to retrieve messages from JMS destinations if they are either:
- listening to non-WebLogic queues; or
- listening to a WebLogic queue and transaction batching is enabled.

When using message-driven beans with foreign destinations we have the following considerations

The concurrency of message-driven beans that consume from destinations hosted by foreign providers (non-WebLogic JMS destinations) is determined using the same algorithm that is used for WebLogic JMS destinations.
Message-Driven beans listening on queues - Instead of a dedicated thread, each deployed message-driven bean uses a token-based, synchronous polling mechanism that always uses at least one thread from the dispatch-policy.

Here token-based message polling works as follows

As of WebLogic 10.3, the polling mechanism changed to a token-based approach to provide better control of the concurrent poller thread count under changing message loads. In previous releases, the thread count ramp-up could be too gradual in certain use cases. Additionally, child pollers, once awoken, could not be ramped down and returned back to the pool for certain foreign JMS providers.
When a thread is returned to the thread pool with token-based polling, the thread's internal JMS consumer is closed rather than cached. This assures that messages will not be implicitly pre-fetched by certain foreign JMS Providers while there is no polling thread servicing the consumer.
In addition, each message-driven bean maintains a single token that provides permission for a given poller thread to create another thread.
- On receipt of a message - A poller thread that already has the token or that is able to acquire the token because the token is not owned, wakes up an additional poller thread and gives the token to the new poller if the maximum concurrency has not yet been reached. If maximum concurrency has been reached, the poller thread simply releases the token (leaving it available to any other poller).
- On finding an empty queue/topic - A poller tries to acquire the token and if successful will try to poll the queue periodically. If it fails to acquire the token, it returns itself back to the pool. This ensures that with an empty queue or topic, there is still at least one poller checking for messages.
WebLogic 12.2.1.2.0 introduces two properties of type activation-configproperty: mdbDestinationPollIntervalMillis and minimizeAQSessions.
- The mdbDestinationPollInternvalMillis property controls the message polling interval that is used by the synchronous polling mechanism for message-driven bean.
- When the JMS provider is AQJMS, the minimizeAQSessions property reduces the use of database resources by minimizing the number of AQ JMS sessions.

Additional considerations are described in the Oracle Support Doc ID 1502885.1

// Reduce sleep time between pools to 10ms and max of 15ms.
// the default =15secs causing messages to be queued up.
-Doracle.jms.minSleepTime=100
-Doracle.jms.maxSleepTime=200

// The parameter weblogic.ejb.container.MDBMessageWaitTime is measured in seconds.
// With Patch 13415672 this can be specified as a float as well.
// The minimum adviced value for this parameter is 250 ms (i.e. -Dweblogic.ejb.container.MDBMessageWaitTime=.25)
-Dweblogic.ejb.container.MDBMessageWaitTime=1

// Prevents simultaneous DB connection attempts by the MDBs (during startup)
-Dweblogic.ejb.container.MDBDestinationPollIntervalMillis=5000

// Free Resources
-Dweblogic.mdb.message.MinimizeAQSessions=true

Oracle Support Doc ID 1319580.1 and Doc ID 388291.1 provide some additional background information considering the above settings:

Non-transactional Message-Driven Beans use a foreign vendor's thread, not a WebLogic Server thread. In this situation, a dispatch-policy is ignored except for determining concurrency. In this case, a polling approach is used as well, i.e., it uses an exponential back off strategy to calculate the sleep time between two consecutive polls when the destination is empty. The sleep time can be controlled by the following system properties -Doracle.jms.minSleepTime and -Doracle.jms.maxSleepTime.
Transactional Message-Driven Beans run in WebLogic Server threads, as follows:
- Message-Driven Beans listening on topics: Each deployed Message-Driven Bean uses a dedicated daemon polling thread that is created in Non-Pooled Threads thread group.
- Message-Driven Beans listening on queues: Instead of a dedicated thread, each deployed Message-Driven Bean uses a token-based, synchronous polling mechanism that always uses at least one thread from a dispatch-policy. In this case the JMSMessagePoller is used, which has a polling latency that defaults to 3 seconds when the queue is empty. This default latency can be controlled with the -Dweblogic.ejb.container.MDBMessageWaitTime system property which is measured in seconds.

For transacted Message-Driven Beans the following setings are advised: -Doracle.jms.minSleepTime=100 -Doracle.jms.maxSleepTime=200 -Dweblogic.ejb.container.MDBMessageWaitTime=1.

References

[1] Oracle Database Online Documentation.
[2] Oracle WebLogic Server.
[3] Java Platform, Enterprise Edition: The Java EE Tutorial.

Fun with Virtualization

2017-09-19T14:20:00.000+02:00

The successful deployment of Software on virtualization platforms is not significantly different from deploying Software on physical servers. We can fully leverage our current skill set while also delivering the benefits associated with virtualization. There are all kinds of documentation out there that can help us leveraging best pratices in virtualization. Below, we present a summary (or checklist) of these best practices.

CPU

Total number of VCPUs assigned to all VMs is equal to or less than the total number of physical cores

Do not over-allocate vCPUs try to match the exact workload. If monitoring of the actual workload shows that the Software is not benefiting from the increased vCPUs, the excess vCPUs impose scheduling constraints and can degrade overall performance of the virtual machine.
For Business Critical Applications ( aka Tier-1-Applications ) the recommendation is to avoid over commitment of processor resources (maintain a 1:1 ratio of physical cores to vCPUs).

A best practice for virtual machines is to right-size them to the amount of CPU and memory that is needed, and to keep the NUMA node boundaries in mind when doing this sizing. In other words, assign the amount of vCPUs and memory that are needed for the VM, but limit this to what is really needed. If the size of the VM can be kept to equal or less than the size of the NUMA node, then performance can be optimized, as remote NUMA node memory access is limited or eliminated. This aspect of NUMA performance was measured on the 4-socket (or 4 NUMA node) Broadwell server. Four 24-vCPU virtual machines were tested and compared against the performance of a single 96-vCPU VM. The results in the chart below show that the total throughput of the four VMs was 15% higher than the throughput of the single VM (see here for more information).

Enable hyperthreading

Hyperthreading enables a single physical processor core to behave like two logical processors, allowing two independent threads to run simultaneously. Unlike having twice as many processor cores which can roughly double performance hyperthreading can provide anywhere from a slight to a significant increase (up to 24 percent) in system performance by keeping the processor pipeline busier.

Avoid CPU affinity on systems with Hyperthreading. Pinning vCPUs from different/single SMP VMs to both logical processors on one core causes poor performance because logical processors share processor resources and that will affect the ability of the NUMA scheduler to rebalance VMs across NUMA nodes for fairness.

Keep VM size to within the NUMA node

Wide VMs are defined as VMs with more vCPUs than the number of cores in each physical NUMA node are referred to as wide VMs. Because vCPUs in these wide VMs might need to access memory outside their own NUMA node, they might experience higher average memory access latencies than virtual machines that fit entirely within a NUMA node. The recommendation is to keep the default of one core per socket (with the number of virtual sockets therefore equal to the number of vCPUs) for vNUMA to match physical NUMA topology. Try to align VMs with physical NUMA boundaries.

Number of virtual sockets versus number of cores per socket

In the blog post Does corespersocket Affect Performance?, a test is presented in which a number of different set-ups are compared. The following items are concluded:

When creating a virtual machine, by default, vSphere will create as many virtual sockets as we have requested vCPUs and the cores per socket is equal to one. We think of this configuration as 'wide' and 'flat'. This will enable vNUMA to select and present the best virtual NUMA topology to the guest operating system, which will be optimal on the underlying physical topology.
When we must change the cores per socket though, commonly due to licensing constraints, ensure we mirror physical server's NUMA topology. This is because when a virtual machine is no longer configured by default as 'wide' and 'flat', vNUMA will not automatically pick the best NUMA configuration based on the physical server, but will instead honor wer configuration - right or wrong - potentially leading to a topology mismatch that does affect performance.

Note that if vNUMA is configured with vCPU hotplug settings, the virtual machine will be started without vNUMA and instead it will use Uniform Memory Access with interleaved memory access (see here for more information).

Memory

Set memory reservations to avoid over-commitment, provide available physical memory

Size memory based on NUMA node size. Overallocation of memory to a virtual machine beyond what is needed can result in crossing physical NUMA node boundaries unnecessarily. Assign the needed memory, but do not over allocate, for example,
- Oracle Database: Set memory reservations equal to the sum total of the size of the Oracle SGA, the Oracle PGA, the Oracle Background processes stack space and Operating System Used Memory. The memory reservation should be large enough to avoid kernel swapping between ESX and the guest OS because Oracle databases can be memory-intensive.
- Oracle WebLogic Server: Set memory reservations equal to the sum of Xmx + 0.5 Xmx (where Xmx is the maximum heap size of each Java Virtual Machine running on the VM) and Operating System Used Memory (see here for more information).

Use HugePages

The use of huge pages results in reduced memory management overhead and can increase hypervisor performance. Huge pages (2MB insteas of 4kB memory pages) improve performance by significantly reducing TLB misses (applications with large active memory working sets).

memory_to_reserve_in_mega_bytes = ...
number_of_huge_pages = memory_to_reserve_in_mega_bytes / 2
mem_lock = 2048 * (memory_to_reserve_in_mega_bytes / 2)


In /etc/sysctl.conf add the following lines

# The percentage of how aggressively memory pages are swapped to disk
vm.swappiness = 0
# The percentage of main memory the pdflush daemon should write data out to the disk.
vm.dirty_background_ratio = 25
# The percentage of main memory the actual disk writes will take place.
vm.dirty_ratio = 20
# set the number of huge pages based on the Hugepagesize, i.e., 2048kB
vm.nr_hugepages = ${number_of_huge_pages}

# give permission to the group that runs the process to access the shared memory segment
# to this end open the /etc/group file and retrieve the group-id
# when the oracle standard is used, the (oinstall) group id will be 54321
vm.hugetlb_shm_group = 54321


In /etc/security/limits.conf add the following lines

# memlock - maximum locked in-memory address space (kB)
@oinstall soft memlock ${mem_lock}
@oinstall hard memlock ${mem_lock}


When using JRockit we have to make a hugetblfs file system available in the directory /mnt/hugepages. 

mkdir -p /mnt/hugepages
chmod -R 777 /mnt/hugepages

In /etc/fstab add the following lines

nodev    /mnt/hugepages    hugetlbfs    defaults,mode=777    0 0

Networking

Separate management traffic

Virtual machines should not see infrastructure traffic (security violation) and should not be impacted by infrastructure traffic bursts (for example, vSphere vMotion operations).

Allocate at least 2 NICs per vSwitch to leverage NIC teaming capabilities

NIC teams can share the load of traffic among some or all of its members, or provide passive failover in the event of a hardware failure or a network outage.

Use the VMXNET3 para-virtualized NIC

The paravirtualized network adapters in the VMXNET family implement an optimized network interface that passes network traffic between the virtual machine and the physical network interface cards with minimal overhead.

Use network I/O control to converge network and storage traffic onto 10GbE

This can reduce cabling requirements, simplify management, and reduce cost.

Storage

Create dedicated datastores to service database workloads. The creation of dedicated datastores for I/O-intensive databases is analogous to provisioning dedicated LUNs in the physical world. This is a typical design for a mission-critical enterprise workload, for example,
- Oracle Database: Use dedicated LUNs or datastores for data and log virtual disks. Ensure that the storage being used for the data and log virtual disks is not shared with other virtual machines or applications. Use paravirtualized SCSI adapters for Oracle datafiles with demanding workloads. The combination of the new paravirtualized SCSI driver (PVSCSI) and additional ESX/ESXi kernel-level storage stack optimizations dramatically improves storage I/O performance.
- Oracle WebLogic Server and then specifically FileStores:
  - For basic (non-RAID) disk hardware, consider dedicating one disk per file store. A store can operate up to four to five times faster if it does not have to compete with any other store on the disk. Remember to consider the existence of the default file store in addition to each configured store and a JMS paging store for each JMS server.
  - If disk performance continues to be a bottleneck, consider purchasing disk or RAID controller hardware that has a built-in write-back cache. These caches significantly improve performance by temporarily storing persistent data in volatile memory. To ensure transactionally safe write-back caches, they must be protected against power outages, host machine failure, and operating system failure. Typically, such protection is provided by a battery-backed write-back cache.

In an older document, an example for a storage lay-out is given for the Oracle Database:

LUN 1 – 8 (1TB each – RAID 5) -> DATA: datastore 1 – 8 VMFS -> 8 x VMDKs (900GB each)
LUN 9 - 10 (1.5TB each – RAID 10) -> REDO: datastore 9 - 10 VMFS -> 2 x VMDKs (1TB each)
LUN 11 (1.5TB – RAID 10) -> VMORA: datastore 11 VMFS -> 2 VMDK (30GB for OS, 50GB for Binary)

In the case of Oracle WebLogic Server, it is in general not necessary to use multiple datastores, i.e., most storage systems have controller hardware and protected write caches:

Write cache is mirrored across controllers, and the system offers RAID 1+0 (mirroring + striping), RAID 5+0 (RAID 5 distributed parity + striping), and RAID MP (multiple distributed parity with striping) across the drives behind the nodes. In addition, controller nodes are configured in logical pairs whereby each node has a partner. The partner nodes have redundant physical connections to the subset of physical drives owned by the node pair. Within the pair, each serves as the backup node for the LDs owned by the partner node. If a controller node were to fail, data availability would be unaffected. In the event of a node failure, that node’s partner takes over the LDs for the failed node. It then immediately flushes data in the write cache on other nodes in the array that belongs on the LDs it has taken over.

Use paravirtualized SCSI adapters

The combination of the new paravirtualized SCSI driver (PVSCSI) and additional ESX/ESXi kernel-level storage stack optimizations dramatically improves storage I/O performance.

Create dedicated datastores to service database workloads

The creation of dedicated datastores for I/O-intensive databases is analogous to provisioning dedicated LUNs in the physical world. This is a typical design for a mission-critical enterprise workload.

Use multiple vSCSI adapters and evenly distribute target devices

When using PVSCSI adapters we have 254 OIPS: "The large-scale workloads with intensive I/O patterns require adapter queue depths greater than the Paravirtual SCSI (PVSCSI) default values. Current PVSCSI queue depth default values are 64 (for device) and 254 (for adapter). We can increase PVSCSI queue depths to 254 (for device) and 1024 (for adapter) inside a Windows virtual machine or Linux virtual machine (see here for more information)." In the case of parallelism "It's worth noting that we can configure a total of 4 vSCSI adapters per virtual machine. To provide the best performance, one should also distribute virtual disks across as many vSCSI adapters as possible. This configuration provides the capability to process more IO simultaneously and benefit from additional queues if necessary (see here for an example - additional information can be found here)".

Use vSphere VMFS

To balance performance and manageability in a virtual environment use VMFS. To store virtual disks, ESX/ESXi uses datastores, which are logical containers that hide specifics of physical storage from virtual machines and provide a uniform model for storing virtual machine files. Datastores that we deploy on block storage devices use the vSphere VMFS format, a special high-performance file system format that is optimized for storing virtual machines.

Align VMFS properly

Like other disk-based file systems, VMFS suffers a penalty when the partition is unaligned. Use VMware vCenter to create VMFS partitions because it automatically aligns the partitions. In an older document some recommendations are given.

VM placement

When a VM is being powered up in a DRS cluster, DRS runs its algorithm to determine the right ESXi host for it to be powered up on. This decision, also known as VM placement (or initial placement) is made based on the expected change in resource distribution (after ensuring that there are no constraint violations if the VM was placed on the host). One of the first steps in ensuring good VM performance is to make sure that the VM gets all the resources it needs as soon as it is powered on. DRS considers the demand of a VM, so it will never be short of resources whenever it is started.

Apart from the fact that "DRS strives to keep out VMs happy, we also need to keep in mind Oracle WebLogic Server Clustering, i.e., making sure server in a cluster are placed on different VMs when necessary. So usually, we have to create a number of rules:

VM/Host Rules:
- Keep Virtual Machines Together (VM-VM) - This rule ensures that the VMs specified in the rule are always running on the same host.
- Separate Virtual Machines (VM-VM) - This rule will keep the VMs in the rule always running on different hosts.
- Virtual Machines to Hosts (VM-Host) - This type of rule is set on groups of one or more VMs and one or more hosts.

And last but not least: "Start with a thorough understanding of your workload".

References

[1] Oracle Databases on VMware Best Practices Guide.
[2] Troubleshooting ESX/ESXi virtual machine performance issues (2001003).
[3] Overview Performance Charts Disk Requests (Number).