My Tech World

Installing Grafana with ingress

2019-10-12T04:17:00.003-07:00

Installing Grafana with ingress

If you are deploying Prometheus and Grafana in Kubernetes, easiest way would be to use helm charts. But when you are deploying the helm chart, You will need to provide custom parameters mainly for storage configuration and service configuration to expose Grafana UI to outsiders.

In this blog I will share the artifacts I used in the AWS EKS. But if someone wants to change it to use in any other platform or a custom installation, just change the ingress annotations and you will be good.

For the basic installation I used eks workshop[1] to start with. Here everything is provided in command line. Instead I used yaml files and used the file with -f option of the helm command. My sample helm command would be like below.

helm install --name promethues stable/prometheus -f prometheus.yaml --namespace prometheus

My prometheus.yaml will be like below. Here I haven’t changed anything from eks workshop other than taking everything to yaml file. But if you intend to change something you can do so by adding more content to this yaml according to the structure of prometheus values.yaml[2].

alertmanager:
  persistentVolume:
    storageClass: "gp2"
server:
  persistentVolume:
    storageClass: "gp2"

Tricky part would be to adding ingress to Grafana installation. I obviously followed the same path. I just customized values according to values.yaml of Grafana Helm chart[3].

Here some important notes are ingress.enabled should set to true. And then ingress.path should set to “/*” support Grafana UI. Other annotations set here are to support ALB ingress controller and to enable SSL to ALB.

And you can customize any other thing following format of the values.yaml of the chart.

persistence:
  storageClassName: "gp2"
adminPassword: "EKS!sAWSome"
ingress:
  enabled: true
  path: /*
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:iam::xxxxxxxxx:server-certificate/aws
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    external-dns.alpha.kubernetes.io/hostname: grafana.xxxx.xxx.xxx
service:
  annotations:
    alb.ingress.kubernetes.io/target-type: ip
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      -  name: Prometheus
         type: prometheus
         url: http://prometheus-server.prometheus.svc.cluster.local
         access: proxy
         isDefault: true

Hope this helps!!!.

References

[1] https://eksworkshop.com/monitoring/

[2] https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml

[3] https://github.com/helm/charts/blob/master/stable/grafana/values.yaml

Installing gluster in AWS EKS

2019-06-03T08:43:00.001-07:00

This article is a continuance of [1]. Purpose of this article is to document the steps, issues and solutions to those issues we have to face when installing gluster in EKS (Elastic Kubernetes Service).

For gluster we need a disk to be attached with the K8s node. In EKS easiest way of implementing this is, adding it to the node configuration. So every time a node comes up, it comes up with a disk attached to the defined path. You can use this path in the topology.josn as mentioned in [1].

Next step is to install gluster using the gk-deploy script. The challenge comes here after. To use gluster in pods, you need to define a storage class. The heketi url mentioned in the storage class definition, should be accessible from master node. But the given heketi url is a cluster IP type k8s service. But in EKS deployments masters are managed by AWS and master don't have access to cluster IPs. So how we can solve this? Actually I tried to contact AWS support on this and I didn't got the answer from them. That lead to finding the answer by my own.

I created a LoadBalancer type k8s service in k8s fronting actual heketi service. First I started with external lb type but that let anyone outside the network access my heketi. Then I tried internal load balancer type. Then again I came to the original place. But this time I was able to make that internal lb type service accessible to the master by adding all my nodes to the same VPC as my master is in. And it allowed me to create the storage class without an issue and I could continue to create the PVC and use it in pods. Definition looks like below.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
  labels:
    glusterfs: heketi-deployment
    heketi: deployment
  name: heketi-external
  namespace: gluster
spec:
  externalTrafficPolicy: Cluster
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    glusterfs: heketi-pod
    heketi: pod
  sessionAffinity: None
  type: LoadBalancer

Hope this helps. Good luck on your deployment.

[1] https://wdfdo1986.blogspot.com/2019/05/consuming-file-system-artifacts-from.html

Consuming File System artifacts from Kubernetes Pods

2019-05-31T10:42:00.001-07:00

When you are deploying an application which contains artifacts written on file system dynamically withing kubernetes (k8s), for example a tomcat server exposed to outside to deploy war files, you need to make sure the file system state is preserved always. Otherwise if the pod goes down, you might loose data.

So one solution is to mount an external disk. Yes indeed you can do that. But how robust is that solution. Say something happened to the external disk. How can you recover the data? Use several disks and rsync to sync the data. Sounds a robust solution. Say you want to increase the reliability. And what happens if rsync process get killed. How much will it cost to make it's reliability closer to 100%?

We have a robust, simple solution. It's using gluster to save data. [1] [2]

We install a pod named gluster for each node. There is an additional disk attached to each node which will be used as the data storage for gluster. This disk is formatted in a special format and attached to the gluster.

This disk can be attached to a specific path and we can mention it as below (eg :- /dev/sdb) in the topology.json. Following is a sample. When your run gkdeploy script to install gluster, you have to place topology.json in the same location.

{
   "clusters":[
      {
         "nodes":[
            {
               "node":{
                  "hostnames":{
                     "manage":[
                        "node0"
                     ],
                     "storage":[
                        "192.168.10.100"
                     ]
                  },
                  "zone":1
               },
               "devices":[
                  "/dev/sdb"
               ]
            },
            {
               "node":{
                  "hostnames":{
                     "manage":[
                        "node1"
                     ],
                     "storage":[
                        "192.168.10.101"
                     ]
                  },
                  "zone":1
               },
               "devices":[
                  "/dev/sdb"
               ]
            },
            {
               "node":{
                  "hostnames":{
                     "manage":[
                        "node2"
                     ],
                     "storage":[
                        "192.168.10.102"
                     ]
                  },
                  "zone":1
               },
               "devices":[
                  "/dev/sdb"
               ]
            }
         ]
      }
   ]
}

Once you install gluster in k8s now comes the question on how they interconnect? There is an another component called heketi [3] which is generally used to manage volumes. Here heketi is used to manage gluster cluster. You can use heketi cli [4] to do advanced operations. Heketi is included in the gk-deploy script from glusterfs.

Note that this blog is written to explain a problem and a solution. Here I haven't included any low level operational details or installation details.

Once gluster is installed in your k8s setup, next question is to how to use it? First you need to create a storage class in k8s with the heketi cli url. Here this url should be accessible to master node of k8s deployment. It looks like below.


	kind: StorageClass
	apiVersion: storage.k8s.io/v1beta1
	metadata:
	name: glusterfs-storage
	provisioner: kubernetes.io/glusterfs
	parameters:
	resturl: "http://${HEKETI_URL}"

After that there should be a pvc (Persisted Volume Claim) to use this storage class.

	apiVersion: v1
	kind: PersistentVolumeClaim
	metadata:
	name: gluster-pvc
	annotations:
	volume.beta.kubernetes.io/storage-class: glusterfs-storage
	spec:
	accessModes:
	- ReadWriteOnce
	resources:
	requests:
	storage: 2Gi

Then you can use pvc in your pod conflagration as a volume mount.

	volumeMounts:
	-
	mountPath: /mnt/files/diff
	name: sample-files

	volumes:
	-
	name: synapse-files
	persistentVolumeClaim:
	claimName: gluster-pvc

Hope this helped someone to create a solution.
Good luck on deployment!!!

[1] https://github.com/gluster/glusterfs
[2] https://github.com/gluster/gluster-kubernetes
[3] https://github.com/heketi/heketi
[4] https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/container-native_storage_for_openshift_container_platform/chap-documentation-red_hat_gluster_storage_container_native_with_openshift_platform-heketi_cli

Generate JWT access tokens from WSO2 Identity Server

2016-11-07T22:25:00.003-08:00

In Identity Server 5.2.0 we have created an interface to generate access tokens. Using that we have developed a sample to generate JWT tokens. You can find that sample under msf4j samples[1][2]. If you are build it as it is you will need to use Java 8 to build since msf4j is developed on Java 8. So you will need to run Identity Server on Java 8 as well. After building the project[2] please copy the jar inside target directory to $IS_HOME/repository/components/dropins/ directory. And then please add the following configuration to Identity.xml which is placed under $IS_HOME/repository/conf/identity/ folder inside tag OAuth.

 <IdentityOAuthTokenGenerator>com.wso2.jwt.token.builder.JWTAccessTokenBuilder</IdentityOAuthTokenGenerator>

Then go to the database you used to store oauth tokens (This is the database pointed from the datasource you mentioned in the $IS_HOME/repository/conf/identity/identity.xml) and then alter the size of the column ACCESS_TOKEN of the table IDN_OAUTH2_ACCESS_TOKEN to the maximum value provided by your database provider.

Now when you generate access tokens, they should be JWTs.

References

[1] https://github.com/wso2/msf4j/tree/master/samples/jwt-claims

[2] https://github.com/wso2/msf4j/tree/master/samples/jwt-claims/JWTAccessTokenBuilder

WSO2 Identity Server 5.2.0 - Setup Multiple Attribute login with JDBC userstore

2016-11-07T07:05:00.001-08:00

In WSO2 Products multiple attribute login (login with either email or username for example) can be done with LDAP Userstore manager with simply by changing some configurations. But with JDBC Userstore manager we need some customization to achieve that. We can achieve that by using Implementing a custom userstore manager. In this blog entry I am going to make work with email and username. You can find the full sample here[1].

For login purposes

To login to the server with multiple attributes, you will need to override doAuthenticate method and doGetExternalRoleListOfUser method. Following are the overridden methods for login.

   @Override  
   public boolean doAuthenticate(String attribute, Object credential) throws UserStoreException {  
     if (!checkUserNameValid(attribute)) {  
       return false;  
     }  
     if (!checkUserPasswordValid(credential)) {  
       return false;  
     }  
     if (UserCoreUtil.isRegistryAnnonymousUser(attribute)) {  
       log.error("Anonnymous user trying to login");  
       return false;  
     }  
     Connection dbConnection = null;  
     ResultSet rs = null;  
     PreparedStatement prepStmt = null;  
     String sqlstmt = null;  
     String password = (String) credential;  
     boolean isAuthed = false;  
     try {  
       dbConnection = getDBConnection();  
       dbConnection.setAutoCommit(false);  
       sqlstmt = realmConfig.getUserStoreProperty(JDBCRealmConstants.SELECT_USER);  
       if (log.isDebugEnabled()) {  
         log.debug(sqlstmt);  
       }  
       prepStmt = dbConnection.prepareStatement(sqlstmt);  
       // Insert attribute as parameter for each occurrence of   
       int paramCount = StringUtils.countMatches(sqlstmt, "?");  
       // If we specify the tenant into query, we assume that it is the last parameter  
       if (sqlstmt.contains(UserCoreConstants.UM_TENANT_COLUMN)) {  
         // Assign attribute value to parameters, except the last one  
         for (int i = 1; i < paramCount; i++) {  
           prepStmt.setString(i, attribute);  
         }  
         prepStmt.setInt(paramCount, tenantId);  
       } else {  
         // There is no tenant indication, set all parameters with attribute value  
         for (int i = 1; i <= paramCount; i++) {  
           prepStmt.setString(i, attribute);  
         }  
       }  
       rs = prepStmt.executeQuery();  
       if (rs.next() == true) {  
         String storedPassword = rs.getString(3);  
         String saltValue = null;  
         if ("true".equalsIgnoreCase(realmConfig  
             .getUserStoreProperty(JDBCRealmConstants.STORE_SALTED_PASSWORDS))) {  
           saltValue = rs.getString(4);  
         }  
         boolean requireChange = rs.getBoolean(5);  
         Timestamp changedTime = rs.getTimestamp(6);  
         GregorianCalendar gc = new GregorianCalendar();  
         gc.add(GregorianCalendar.HOUR, -24);  
         Date date = gc.getTime();  
         if (requireChange == true && changedTime.before(date)) {  
           isAuthed = false;  
         } else {  
           password = this.preparePassword(password, saltValue);  
           if ((storedPassword != null) && (storedPassword.equals(password))) {  
             isAuthed = true;  
           }  
         }  
       }  
     } catch (SQLException e) {  
       String msg = "Error occurred while retrieving user authentication info.";  
       log.error(msg, e);  
       throw new UserStoreException("Authentication Failure");  
     } finally {  
       DatabaseUtil.closeAllConnections(dbConnection, rs, prepStmt);  
     }  
     if (log.isDebugEnabled()) {  
       log.debug("User " + attribute + " login attempt. Login success :: " + isAuthed);  
     }  
     return isAuthed;  
   }  
   @Override  
   public Date getPasswordExpirationTime(String attribute) throws UserStoreException {  
     Connection dbConnection = null;  
     ResultSet rs = null;  
     PreparedStatement prepStmt = null;  
     String sqlstmt;  
     Date date = null;  
     try {  
       dbConnection = getDBConnection();  
       dbConnection.setAutoCommit(false);  
       sqlstmt = realmConfig.getUserStoreProperty(JDBCRealmConstants.SELECT_USER);  
       if (log.isDebugEnabled()) {  
         log.debug(sqlstmt);  
       }  
       prepStmt = dbConnection.prepareStatement(sqlstmt);  
       // Insert attribute as parameter for each occurrence of   
       int paramCount = StringUtils.countMatches(sqlstmt, "?");  
       // If we specify the tenant into query, we assume that it is the last parameter  
       if (sqlstmt.contains(UserCoreConstants.UM_TENANT_COLUMN)) {  
         // Assign attribute value to parameters, except the last one  
         for (int i = 1; i < paramCount; i++) {  
           prepStmt.setString(i, attribute);  
         }  
         prepStmt.setInt(paramCount, tenantId);  
       } else {  
         // There is no tenant indication, set all parameters with attribute value  
         for (int i = 1; i <= paramCount; i++) {  
           prepStmt.setString(i, attribute);  
         }  
       }  
       rs = prepStmt.executeQuery();  
       if (rs.next() == true) {  
         boolean requireChange = rs.getBoolean(5);  
         Timestamp changedTime = rs.getTimestamp(6);  
         if (requireChange) {  
           GregorianCalendar gc = new GregorianCalendar();  
           gc.setTime(changedTime);  
           gc.add(GregorianCalendar.HOUR, 24);  
           date = gc.getTime();  
         }  
       }  
     } catch (SQLException e) {  
       String msg = "Error occurred while retrieving password expiration time.";  
       log.error(msg, e);  
       throw new UserStoreException(msg, e);  
     } finally {  
       DatabaseUtil.closeAllConnections(dbConnection, rs, prepStmt);  
     }  
     return date;  
   }  
   public String[] doGetExternalRoleListOfUser(String userName, String filter) throws UserStoreException {  
     if(log.isDebugEnabled()) {  
       log.debug("Getting roles of user: " + userName + " with filter: " + filter);  
     }  
     String sqlStmt;  
     if(this.isCaseSensitiveUsername()) {  
       sqlStmt = this.realmConfig.getUserStoreProperty("UserRoleSQL");  
     } else {  
       sqlStmt = this.realmConfig.getUserStoreProperty("UserRoleSQLCaseInsensitive");  
     }  
     ArrayList roles = new ArrayList();  
     if(sqlStmt == null) {  
       throw new UserStoreException("The sql statement for retrieving user roles is null");  
     } else {  
       String[] names;  
       if(sqlStmt.contains("UM_TENANT_ID")) {  
         names = this.getStringValuesFromDatabase(sqlStmt, new Object[]{userName, userName, Integer.valueOf(this.tenantId), Integer.valueOf(this.tenantId), Integer.valueOf(this.tenantId), Integer.valueOf(this.tenantId)});  
       } else {  
         names = this.getStringValuesFromDatabase(sqlStmt, new Object[]{userName});  
       }  
       if(log.isDebugEnabled()) {  
         if(names != null) {  
           String[] arr$ = names;  
           int len$ = names.length;  
           for(int i$ = 0; i$ < len$; ++i$) {  
             String name = arr$[i$];  
             log.debug("Found role: " + name);  
           }  
         } else {  
           log.debug("No external role found for the user: " + userName);  
         }  
       }  
       Collections.addAll(roles, names);  
       return (String[])roles.toArray(new String[roles.size()]);  
     }  
   }

And with this you will need to modify your $CARBON_HOME/repository/conf/user-mgt.xml user store manager configuration section as below.

     <UserStoreManager class="org.wso2.carbon.userstore.jdbc.CustomJDBCUserStoreManager">  
       <Property name="TenantManager">org.wso2.carbon.user.core.tenant.JDBCTenantManager</Property>  
       <Property name="ReadOnly">false</Property>  
       <Property name="ReadGroups">true</Property>  
       <Property name="WriteGroups">true</Property>  
       <Property name="UsernameJavaRegEx">^[\S]{3,30}$</Property>  
       <Property name="UsernameJavaScriptRegEx">[a-zA-Z0-9@._-|//]{3,30}$</Property>  
       <Property name="UsernameWithEmailJavaScriptRegEx">[a-zA-Z0-9@._-|//]{3,30}$</Property>  
       <Property name="UsernameJavaRegExViolationErrorMsg">Username pattern policy violated</Property>  
       <Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>  
       <Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>  
       <Property name="PasswordJavaRegExViolationErrorMsg">Password length should be within 5 to 30 characters</Property>  
       <Property name="RolenameJavaRegEx">^[\S]{3,255}$</Property>  
       <Property name="RolenameJavaScriptRegEx">^[\S]{3,255}$</Property>  
       <Property name="CaseInsensitiveUsername">true</Property>  
       <Property name="SCIMEnabled">false</Property>  
       <Property name="IsBulkImportSupported">false</Property>  
       <Property name="PasswordDigest">SHA-256</Property>  
       <Property name="StoreSaltedPassword">true</Property>  
       <Property name="MultiAttributeSeparator">,</Property>  
       <Property name="MaxUserNameListLength">100</Property>  
       <Property name="MaxRoleNameListLength">100</Property>  
       <Property name="UserRolesCacheEnabled">true</Property>  
       <Property name="UserNameUniqueAcrossTenants">false</Property>  
       <Property name="PasswordHashMethod">SHA</Property>  
       <Property name="SelectUserSQL">SELECT distinct u.* FROM UM_USER u left join UM_USER_ATTRIBUTE ua on u.UM_ID = ua.UM_USER_ID WHERE u.UM_USER_NAME = ? OR (ua.UM_ATTR_NAME = "mail" AND ua.UM_ATTR_VALUE = ?) AND u.UM_TENANT_ID = ?</Property>  
       <Property name="UserRoleSQLCaseInsensitive">SELECT UM_ROLE_NAME FROM UM_USER_ROLE, UM_ROLE, UM_USER WHERE LOWER(UM_USER.UM_USER_NAME) IN (SELECT LCASE(u.UM_USER_NAME) FROM UM_USER u left join UM_USER_ATTRIBUTE ua on u.UM_ID = ua.UM_USER_ID WHERE u.UM_USER_NAME = ? OR (ua.UM_ATTR_NAME = "mail" AND ua.UM_ATTR_VALUE = ?) AND u.UM_TENANT_ID = ? GROUP BY u.UM_USER_NAME) AND UM_USER.UM_ID=UM_USER_ROLE.UM_USER_ID AND UM_ROLE.UM_ID=UM_USER_ROLE.UM_ROLE_ID AND UM_USER_ROLE.UM_TENANT_ID = ? AND UM_ROLE.UM_TENANT_ID = ? AND UM_USER.UM_TENANT_ID = ?</Property>  
     </UserStoreManager>

To use User Info endpoint with Oauth

Further extending this If you need to use User Info endpoint with oauth2, then you will need to furthe extend following method as well.

   public boolean doCheckExistingUser(String userName) throws UserStoreException {  
     String sqlStmt;  
     if(this.isCaseSensitiveUsername()) {  
       sqlStmt = this.realmConfig.getUserStoreProperty("IsUserExistingSQL");  
     } else {  
       sqlStmt = this.realmConfig.getUserStoreProperty("IsUserExistingSQLCaseInsensitive");  
     }  
     if(sqlStmt == null) {  
       throw new UserStoreException("The sql statement for is user existing null");  
     } else {  
       boolean isExisting = false;  
       String isUnique = this.realmConfig.getUserStoreProperty("UserNameUniqueAcrossTenants");  
       if(Boolean.parseBoolean(isUnique) && !"wso2.anonymous.user".equals(userName)) {  
         String uniquenesSql;  
         if(this.isCaseSensitiveUsername()) {  
           uniquenesSql = this.realmConfig.getUserStoreProperty("UserNameUniqueAcrossTenantsSQL");  
         } else {  
           uniquenesSql = this.realmConfig.getUserStoreProperty("UserNameUniqueAcrossTenantsSQLCaseInsensitive");  
         }  
         isExisting = this.isValueExisting(uniquenesSql, (Connection)null, new Object[]{userName});  
         if(log.isDebugEnabled()) {  
           log.debug("The username should be unique across tenants.");  
         }  
       } else if(sqlStmt.contains("UM_TENANT_ID")) {  
         isExisting = this.isValueExisting(sqlStmt, (Connection)null, new Object[]{userName, userName, Integer.valueOf(this.tenantId)});  
       } else {  
         isExisting = this.isValueExisting(sqlStmt, (Connection)null, new Object[]{userName});  
       }  
       return isExisting;  
     }  
   }

And you will need to add more configurations and following is the updated user store manager configuration.

     <UserStoreManager class="org.wso2.sample.userstore.jdbc.CustomJDBCUserStoreManager">  
       <Property name="TenantManager">org.wso2.carbon.user.core.tenant.JDBCTenantManager</Property>  
       <Property name="ReadOnly">false</Property>  
       <Property name="ReadGroups">true</Property>  
       <Property name="WriteGroups">true</Property>  
       <Property name="UsernameJavaRegEx">^[\S]{3,30}$</Property>  
       <Property name="UsernameJavaScriptRegEx">[a-zA-Z0-9@._-|//]{3,30}$</Property>  
       <Property name="UsernameWithEmailJavaScriptRegEx">[a-zA-Z0-9@._-|//]{3,30}$</Property>  
       <Property name="UsernameJavaRegExViolationErrorMsg">Username pattern policy violated</Property>  
       <Property name="PasswordJavaRegEx">^[\S]{5,30}$</Property>  
       <Property name="PasswordJavaScriptRegEx">^[\S]{5,30}$</Property>  
       <Property name="PasswordJavaRegExViolationErrorMsg">Password length should be within 5 to 30 characters</Property>  
       <Property name="RolenameJavaRegEx">^[\S]{3,255}$</Property>  
       <Property name="RolenameJavaScriptRegEx">^[\S]{3,255}$</Property>  
       <Property name="CaseInsensitiveUsername">true</Property>  
       <Property name="SCIMEnabled">false</Property>  
       <Property name="IsBulkImportSupported">false</Property>  
       <Property name="PasswordDigest">SHA-256</Property>  
       <Property name="StoreSaltedPassword">true</Property>  
       <Property name="MultiAttributeSeparator">,</Property>  
       <Property name="MaxUserNameListLength">100</Property>  
       <Property name="MaxRoleNameListLength">100</Property>  
       <Property name="UserRolesCacheEnabled">true</Property>  
       <Property name="UserNameUniqueAcrossTenants">false</Property>  
       <Property name="PasswordHashMethod">SHA</Property>  
       <Property name="SelectUserSQL">SELECT distinct u.* FROM UM_USER u left join UM_USER_ATTRIBUTE ua on u.UM_ID = ua.UM_USER_ID WHERE u.UM_USER_NAME = ? OR (ua.UM_ATTR_NAME = "mail" AND ua.UM_ATTR_VALUE = ?) AND u.UM_TENANT_ID = ?</Property>  
       <Property name="UserRoleSQLCaseInsensitive">SELECT UM_ROLE_NAME FROM UM_USER_ROLE, UM_ROLE, UM_USER WHERE LOWER(UM_USER.UM_USER_NAME) IN (SELECT LCASE(u.UM_USER_NAME) FROM UM_USER u left join UM_USER_ATTRIBUTE ua on u.UM_ID = ua.UM_USER_ID WHERE u.UM_USER_NAME = ? OR (ua.UM_ATTR_NAME = "mail" AND ua.UM_ATTR_VALUE = ?) AND u.UM_TENANT_ID = ? GROUP BY u.UM_USER_NAME) AND UM_USER.UM_ID=UM_USER_ROLE.UM_USER_ID AND UM_ROLE.UM_ID=UM_USER_ROLE.UM_ROLE_ID AND UM_USER_ROLE.UM_TENANT_ID = ? AND UM_ROLE.UM_TENANT_ID = ? AND UM_USER.UM_TENANT_ID = ?</Property>  
       <Property name="GetUserPropertiesForProfileSQLCaseInsensitive">SELECT UM_ATTR_NAME, UM_ATTR_VALUE FROM UM_USER_ATTRIBUTE, UM_USER WHERE (UM_USER.UM_ID = UM_USER_ATTRIBUTE.UM_USER_ID OR (UM_USER_ATTRIBUTE.UM_ATTR_NAME = 'mail' AND LOWER(UM_USER_ATTRIBUTE.UM_ATTR_VALUE) = LOWER(?))) AND UM_PROFILE_ID=? AND UM_USER_ATTRIBUTE.UM_TENANT_ID=? AND UM_USER.UM_TENANT_ID=?</Property>  
       <Property name="IsUserExistingSQLCaseInsensitive">SELECT distinct u.UM_ID FROM UM_USER u left join UM_USER_ATTRIBUTE ua on u.UM_ID = ua.UM_USER_ID WHERE u.UM_USER_NAME = ? OR (ua.UM_ATTR_NAME = "mail" AND ua.UM_ATTR_VALUE = ?) AND u.UM_TENANT_ID = ?</Property>  
     </UserStoreManager>

And further in $IS_HOME/repository/conf/identity/application-authentication.xml you will need to add following property under Authenticator Config for Basic Authenticator

 <Parameter name="UserNameAttributeClaimUri">http://wso2.org/claims/username</Parameter>

So my Basic Authenticator Config tag is as below.

         <AuthenticatorConfig name="BasicAuthenticator" enabled="true">  
             <Parameter name="UserNameAttributeClaimUri">http://wso2.org/claims/username</Parameter>  
             <!--Parameter name="showAuthFailureReason">true</Parameter-->  
         </AuthenticatorConfig>

With this you will be able to get configured claims when you logged in using different attributes.

References

[1] https://drive.google.com/file/d/0ByTCb2KmTk76dWMwcHMzbWJmVzA/view?usp=sharing

Improving the new relic handler to push api input and output Parameters

2016-11-02T22:49:00.000-07:00

In previous post I described how to publish API traffic to new relic[1]. I have done some modifications to the handler to cater $subject. You will need to change the code to match your exact requirements. And I did some improvements to the code as well. Please find my modified code attachment[2].

I have added following code segments to the handler let me describe of each segment.

Following segment is to build message when POST type is used. Otherwise inside handler we will get an empty envelop.

 try {  
   RelayUtils.buildMessage(((Axis2MessageContext)messageContext).getAxis2MessageContext());  
 } catch (Exception e) {  
   log.warn("Error occured while building message");  
   if(log.isDebugEnabled()){  
     log.debug("Exception thrown while building message", e);  
   }  
 }

Here I push data to new relic, each parameter as a different column. So please change the code as it match your requirement.

Following code segment will extract post data from the request. But this to work properly you will need to set content-type in requests and also backend responses.

 try {  
   OMElement bodyFirstElement = messageContext.getEnvelope().getBody().getFirstElement();  
   if (bodyFirstElement != null) {  
     Iterator iterator = bodyFirstElement.getChildElements();  
     while (iterator.hasNext()) {  
       OMElement element = iterator.next();  
       eventAttributes.put(element.getLocalName(), element.getText());  
     }  
   }   
 } catch (Exception e){  
   log.warn("Error occured while extracting data form the request, hense ignoring");  
   if(log.isDebugEnabled()){  
     log.debug("Exception thrown while extracting data from request", e);  
   }  
 }

Following code segment will capture the query params you provide the api.

 try {  
   Object urlPostfix = ((Axis2MessageContext) messageContext).getAxis2MessageContext().getProperty("REST_URL_POSTFIX");  
   else if (urlPostfix != null) {  
     String[] urlParts = urlPostfix.toString().split("\\?");  
     if (urlParts.length > 0) {  
       String queryParamString = urlParts[1];  
       String[] pairs = queryParamString.split("&");  
       for (String pair : pairs) {  
         String[] keyValues = pair.split("=");  
         if (keyValues.length > 0) {  
           eventAttributes.put(keyValues[0], keyValues[1]);  
         } else {  
           eventAttributes.put(keyValues[0], "");  
         }  
       }  
     }  
   }  
 } catch (Exception e){  
   log.warn("Error occured while extracting data form the request, hense ignoring");  
   if(log.isDebugEnabled()){  
     log.debug("Exception thrown while extracting data from request", e);  
   }  
 }

Good Luck!!! :)

References

[1] http://wdfdo1986.blogspot.com/2016/10/integrate-new-relic-with-wso2-api.html

[2] https://drive.google.com/file/d/0ByTCb2KmTk76M1FabUk1OC0yY00/view?usp=sharing

Integrate New Relic with WSO2 API Manager

2016-10-28T01:47:00.002-07:00

In WSO2 API Manager, we have two transports. HTTP servlet transport and Passthru / NIO transport. All the web application requests are handled through HTTP servlet transport which is on 9763 port and 9443 port with ssl and here we are using tomcat inside WSO2 products. All the service requests are served via Passthru / NIO transport which is on 8082 and 8243 with ssl. When we integrate API Manager with new relic in the way discussed in blog posts [5],[6], new relic only detects the calls made to tomcat transports. So we couldn’t get the API calls related data OOTB.

But by further analyzing new relic APIs I managed to find a workaround for this problem. New relic supports publishing custom events via their insights api[1]. So what we can do is publish these data via custom API handler[2]. Following is a sample implementation of a handler that I used to test the scenario. I will attach the full project herewith[7]. I have created an osgi bundle with this implementation so after building the jar with maven you can drop into $APIM_HOME/repository/components/dropins directory. And make sure you do the necessary changes to this class to match your scenario.

 package org.wso2.carbon.custom.handler.newrelic;  
 import com.newrelic.api.agent.Agent;  
 import com.newrelic.api.agent.Trace;  
 import org.apache.commons.logging.Log;  
 import org.apache.commons.logging.LogFactory;  
 import org.apache.synapse.MessageContext;  
 import org.apache.synapse.rest.AbstractHandler;  
 import java.util.HashMap;  
 import java.util.Map;  
 import com.newrelic.api.agent.NewRelic;  
 public class NewRelicHandler extends AbstractHandler {  
   private static final Log log = LogFactory.getLog(NewRelicHandler.class);  
   @Trace  
   public boolean handleRequest(MessageContext messageContext) {  
     String endpoint = messageContext.getTo().getAddress();  
     Agent agent = NewRelic.getAgent();  
     Map<String, Object> eventAttributes = new HashMap<String, Object>();  
     eventAttributes.put("appId",32034852);  
     eventAttributes.put("appName","APIM");  
     eventAttributes.put("host","localhost");  
     eventAttributes.put("name",endpoint);  
     eventAttributes.put("transactionType","Web");  
     eventAttributes.put("type","Transaction");  
     eventAttributes.put("realAgentId",32034854);  
     eventAttributes.put("transactionSubType","API Call");  
     agent.getInsights().recordCustomEvent("API_calls", eventAttributes);  
     return true;  
   }  
   public boolean handleResponse(MessageContext messageContext) {  
     //TODO: handle response path  
     return true;  
   }  
 }

Since you need engage this handler to all apis you need add following to velocity template as mentioned in the documentation[3]. But for existing apis you will need to add this via source view as mentioned in documentation[3]. Or you can do the same by republishing all existing apis. You will need to add this to both

 $APIM_HOME/repository/resources/api_templates/velocity_template.xml

and

 $APIM_HOME/repository/resources/api_templates/velocity_template.xml

files since this will be needed in default api as well.

 <handler class="org.wso2.carbon.custom.handler.newrelic.NewRelicHandler" />

And further to execute this I added following to the newrelic.yml file.

  custom_insights_events.max_samples_stored: 5000  
  custom_insights_events.enabled: true

Now create publish and subscribe an api (if you haven’t done already) and invoke the api several times. Then go to [4] and create a new dashboard with query

 SELECT * FROM API_calls

You will be able to see API invocation data there. Good Luck :).

References

[1] https://docs.newrelic.com/docs/insights/new-relic-insights/custom-events/insert-custom-events-insights-api
[2] https://docs.wso2.com/display/AM1100/Writing+Custom+Handlers
[3] https://docs.wso2.com/display/AM1100/Writing+Custom+Handlers#WritingCustomHandlers-Engagingthecustomhandler
[4] https://insights.newrelic.com/
[5] http://blog.lasindu.com/2016/07/wso2-application-and-server-monitoring.html
[6] https://scktech.blogspot.com/2016/08/integrate-wso2-products-with-new-relic.html

[7] https://drive.google.com/file/d/0ByTCb2KmTk76MnRfWEI2TElyZkE/view?usp=sharing

Setting up Single node Kubernetes Cluster with Core OS bare metal

2016-08-05T01:43:00.003-07:00

You might know already there is an official documentation to follow to setup a Kubernetes cluster on Core OS bare metal. But when do that specially single node cluster, I found some gaps in that documentation [1]. And another reason for this blog post is to get everything into one place. So this blog post will describe how to overcome the issues of setting up a single node cluster.

Installing Core OS bare metal.

You can refer to doc [2] to install core os.

First thing is about users. Documentation[2] tells you how to create a user without password. To login as that user you will need ssh keys. So to create a user with username password, you can use a cloud-config.yaml file. Here is a sample.

 #cloud-config  
 users:  
 - name: user  
  passwd: $6$SALT$3MUMz4cNIRjQ/Knnc3gXjJLV1vdwFs2nLvh//nGtEh/.li04NodZJSfnc4jeCVHd7kKHGnq5MsenN.tO6Z.Cj/  
  groups:  
  - sudo  
  - docker

Here value for passwd is a hash value. One of the below methods can be used to hash a password.[3]

 # On Debian/Ubuntu (via the package "whois")  
 mkpasswd --method=SHA-512 --rounds=4096  
 # OpenSSL (note: this will only make md5crypt. While better than plantext it should not be considered fully secure)  
 openssl passwd -1  
 # Python (change password and salt values)  
 python -c "import crypt, getpass, pwd; print crypt.crypt('password', '\$6\$SALT\$')"  
 # Perl (change password and salt values)  
 perl -e 'print crypt("password","\$6\$SALT\$") . "\n"'

If you are installing this inside a private network (office network or university network) then you may need to set IP, DNS and so on. Specially about DNS, since its resolving using resolv.conf and it get replaced always then you may need to set it up as below.

Create a file in /etc/systemd/network/static.network with content below. Replace the values with your network values.

 [Match]  
 Name=enp2s0  
 [Network]  
 Address=x.x.x.x  
 Gateway=x.x.x.x  
 DNS=x.x.x.x

Then restart the network with command below.

 sudo systemctl restart systemd-networkd

Now your core os installation is ready to install Kubernetes.

Installing Kubernetes on Core OS

The official documentation [1] describes how to install a cluster. But what I will explain is to create a single node cluster. You can follow the same documentation. When you create certs create what's needed for the master node. And then go on and deploy the master node. Here you will not need the calcio related steps if you don't need to specifically use calcio with Kubernetes.

In Core OS Kubernetes is installed as a service named kubelet. So there what you defined is the service definition and supporting manifest files to service. There are four components of Kubernetes which are configured as manifests inside /etc/kubernetes/manifests/

API server
Proxy
Controller Manager
Scheduler

All these four components will start as pods / containers inside the cluster.

Apart from these four configuration you have configured the kubelet service as well.

But with only these configurations if you try to create a pod it will not get created. Actually it will fail to schedule. Because you don't have a node available to schedule in the cluster. Usually masters don't schedule pods. So that's why in this documentation in master scheduling is set to false. So to turn on scheduling just edit the service definition file /etc/kubernetes/system/kubelet.service to change --register-schedulable=false to --register-schedulable=true .

Now you will be able to schedule the pods in this node.

Configuring to use registry.

Next step is configuring to use a registry. If you have already used docker in other OS, then you should know that adding an insecure registry is done using DOCKER_OPTS. One way to configure DOCKER_OPTS in Core OS is to add it to /run/flannel_docker_opts.env file. But it would be overridden when the server is restarted. For both insecure and proper registries use the method explained in [4].

References

[1] https://coreos.com/kubernetes/docs/latest/getting-started.html

[2] https://coreos.com/os/docs/latest/installing-to-disk.html
[3] https://coreos.com/os/docs/latest/cloud-config.html
[4] https://coreos.com/os/docs/latest/registry-authentication.html

Create an application in WSO2 App Cloud using Maven Plugins

2016-05-05T20:19:00.002-07:00

In Application Development life cycle continuous integration is an important factor. How easy to get something deployed which is built in a build server. You can simply use maven exec plugin to run Curl commands to call rest apis.

Following is an example. Before call the create application api we need to call login api and get created a logged in session. To do that we need to call login api with -c cookies and we need to call create application api with -b cookies.

       <plugin>  
         <groupId>org.codehaus.mojo</groupId>  
         <artifactId>exec-maven-plugin</artifactId>  
         <version>1.2</version>  
         <executions>  
           <execution>  
             <id>login</id>  
             <phase>deploy</phase>  
             <goals>  
               <goal>exec</goal>  
             </goals>  
             <configuration>  
               <executable>curl</executable>  
               <arguments>  
                 <argument>-v</argument>  
                 <argument>-k</argument>  
                 <argument>-c</argument>  
                 <argument>cookies</argument>  
                 <argument>-X</argument>  
                 <argument>POST</argument>  
                 <argument>-F</argument>  
                 <argument>action=login</argument>  
                 <argument>-F</argument>  
                 <argument>userName=<email @ replaced with .>@<tenant domain></argument>  
                 <argument>-F</argument>  
                 <argument>password=<password></argument>  
                 <argument>https://newapps.cloud.wso2.com/appmgt/site/blocks/user/login/ajax/login.jag</argument>  
               </arguments>  
             </configuration>  
           </execution>  
           <execution>  
             <id>create application</id>  
             <phase>deploy</phase>  
             <goals>  
               <goal>exec</goal>  
             </goals>  
             <configuration>  
               <executable>curl</executable>  
               <arguments>  
                 <argument>-v</argument>  
                 <argument>-k</argument>  
                 <argument>-b</argument>  
                 <argument>cookies</argument>  
                 <argument>-X</argument>  
                 <argument>POST</argument>  
                 <argument>https://newapps.cloud.wso2.com/appmgt/site/blocks/application/application.jag</argument>  
                 <argument>-F</argument>  
                 <argument>action=createApplication</argument>  
                 <argument>-F</argument>  
                 <argument>applicationName=Buzzwords&#x20;Backend</argument>  
                 <argument>-F</argument>  
                 <argument>applicationDescription=API&#x20;Producer&#x20;application&#x20;for&#x20;buzzword&#x20;sample</argument>  
                 <argument>-F</argument>  
                 <argument>conSpecMemory=512</argument>  
                 <argument>-F</argument>  
                 <argument>conSpecCpu=300</argument>  
                 <argument>-F</argument>  
                 <argument>runtime=2</argument>  
                 <argument>-F</argument>  
                 <argument>appTypeName=mss</argument>  
                 <argument>-F</argument>  
                 <argument>applicationRevision=${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.nextIncrementalVersion}</argument>  
                 <argument>-F</argument>  
                 <argument>uploadedFileName=${artifactId}-${version}.jar</argument>  
                 <argument>-F</argument>  
                 <argument>runtimeProperties=runtimeProperties=[{"key":"k1","value":"e1"}]</argument>  
                 <argument>-F</argument>  
                 <argument>tags=[{"key":"k1","value":"t1"}]</argument>  
                 <argument>-F</argument>  
                 <argument>fileupload=@${project.build.directory}/${artifactId}-${version}.jar</argument>  
                 <argument>-F</argument>  
                 <argument>isFileAttached=true</argument>  
                 <argument>-F</argument>  
                 <argument>isNewVersion=true</argument>  
               </arguments>  
             </configuration>  
           </execution>  
         </executions>  
       </plugin>

You don't have to deploy it each time you build it. So you can set the phase of the executions to deploy as above. But it might try to deploy the artifact to nexus. To stop that you can skip deploy by adding following.

       <plugin>  
         <artifactId>maven-deploy-plugin</artifactId>  
         <version>2.7</version>  
         <configuration>  
           <skip>true</skip>  
         </configuration>  
       </plugin>

In App Cloud to deploy the changes we need to create new version. So to do that we will always need to increase the version name of the create request. You can use helper plugin and replace plugin as a combination. With following configuration I am creating a property and I am replacing them in each deploy with next patch version number.

       <plugin>  
         <groupId>org.codehaus.mojo</groupId>  
         <artifactId>build-helper-maven-plugin</artifactId>  
         <version>1.10</version>  
         <executions>  
           <execution>  
             <phase>deploy</phase>  
             <id>parse-version</id>  
             <goals>  
               <goal>parse-version</goal>  
             </goals>  
             <configuration>  
               <versionString>${appcloud.version}</versionString>  
             </configuration>  
           </execution>  
         </executions>  
       </plugin>  
       <plugin>  
         <groupId>com.google.code.maven-replacer-plugin</groupId>  
         <artifactId>replacer</artifactId>  
         <version>1.5.3</version>  
         <executions>  
           <execution>  
             <phase>deploy</phase>  
             <goals>  
               <goal>replace</goal>  
             </goals>  
           </execution>  
         </executions>  
         <configuration>  
           <file>pom.xml</file>  
           <replacements>  
             <replacement>  
               <token>${appcloud.version}</token>  
               <value>${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.nextIncrementalVersion}</value>  
             </replacement>  
           </replacements>  
         </configuration>  
       </plugin>

And you need to have a property like below as well.

   <properties>  
     <appcloud.version>1.0.7</appcloud.version>  
   </properties>

Rest of the details of the apis can be found in [1]. Following is the full build tag and the properties tag in the pom.xml. If you run mvn clean install this would not get triggered. This will only trigger when you run mvn deploy.

 <build>  
     <plugins>  
       <plugin>  
         <groupId>org.codehaus.mojo</groupId>  
         <artifactId>build-helper-maven-plugin</artifactId>  
         <version>1.10</version>  
         <executions>  
           <execution>  
             <phase>deploy</phase>  
             <id>parse-version</id>  
             <goals>  
               <goal>parse-version</goal>  
             </goals>  
             <configuration>  
               <versionString>${appcloud.version}</versionString>  
             </configuration>  
           </execution>  
         </executions>  
       </plugin>  
       <plugin>  
         <groupId>com.google.code.maven-replacer-plugin</groupId>  
         <artifactId>replacer</artifactId>  
         <version>1.5.3</version>  
         <executions>  
           <execution>  
             <phase>deploy</phase>  
             <goals>  
               <goal>replace</goal>  
             </goals>  
           </execution>  
         </executions>  
         <configuration>  
           <file>pom.xml</file>  
           <replacements>  
             <replacement>  
               <token>${appcloud.version}</token>  
               <value>${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.nextIncrementalVersion}</value>  
             </replacement>  
           </replacements>  
         </configuration>  
       </plugin>  
       <plugin>  
         <groupId>org.codehaus.mojo</groupId>  
         <artifactId>exec-maven-plugin</artifactId>  
         <version>1.2</version>  
         <executions>  
           <execution>  
             <id>login</id>  
             <phase>deploy</phase>  
             <goals>  
               <goal>exec</goal>  
             </goals>  
             <configuration>  
               <executable>curl</executable>  
               <arguments>  
                 <argument>-v</argument>  
                 <argument>-k</argument>  
                 <argument>-c</argument>  
                 <argument>cookies</argument>  
                 <argument>-X</argument>  
                 <argument>POST</argument>  
                 <argument>-F</argument>  
                 <argument>action=login</argument>  
                 <argument>-F</argument>  
                 <argument>userName=<email @ replaced with .>@<tenant domain></argument>  
                 <argument>-F</argument>  
                 <argument>password=<password></argument>  
                 <argument>https://newapps.cloud.wso2.com/appmgt/site/blocks/user/login/ajax/login.jag</argument>  
               </arguments>  
             </configuration>  
           </execution>  
           <execution>  
             <id>create application</id>  
             <phase>deploy</phase>  
             <goals>  
               <goal>exec</goal>  
             </goals>  
             <configuration>  
               <executable>curl</executable>  
               <arguments>  
                 <argument>-v</argument>  
                 <argument>-k</argument>  
                 <argument>-b</argument>  
                 <argument>cookies</argument>  
                 <argument>-X</argument>  
                 <argument>POST</argument>  
                 <argument>https://newapps.cloud.wso2.com/appmgt/site/blocks/application/application.jag</argument>  
                 <argument>-F</argument>  
                 <argument>action=createApplication</argument>  
                 <argument>-F</argument>  
                 <argument>applicationName=Buzzwords&#x20;Backend</argument>  
                 <argument>-F</argument>  
                 <argument>applicationDescription=API&#x20;Producer&#x20;application&#x20;for&#x20;buzzword&#x20;sample</argument>  
                 <argument>-F</argument>  
                 <argument>conSpecMemory=512</argument>  
                 <argument>-F</argument>  
                 <argument>conSpecCpu=300</argument>  
                 <argument>-F</argument>  
                 <argument>runtime=2</argument>  
                 <argument>-F</argument>  
                 <argument>appTypeName=mss</argument>  
                 <argument>-F</argument>  
                 <argument>applicationRevision=${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.nextIncrementalVersion}</argument>  
                 <argument>-F</argument>  
                 <argument>uploadedFileName=${artifactId}-${version}.jar</argument>  
                 <argument>-F</argument>  
                 <argument>runtimeProperties=runtimeProperties=[{"key":"k1","value":"e1"}]</argument>  
                 <argument>-F</argument>  
                 <argument>tags=[{"key":"k1","value":"t1"}]</argument>  
                 <argument>-F</argument>  
                 <argument>fileupload=@${project.build.directory}/${artifactId}-${version}.jar</argument>  
                 <argument>-F</argument>  
                 <argument>isFileAttached=true</argument>  
                 <argument>-F</argument>  
                 <argument>isNewVersion=true</argument>  
               </arguments>  
             </configuration>  
           </execution>  
         </executions>  
       </plugin>  
       <plugin>  
         <artifactId>maven-deploy-plugin</artifactId>  
         <version>2.7</version>  
         <configuration>  
           <skip>true</skip>  
         </configuration>  
       </plugin>  
     </plugins>  
   </build>  
   <properties>  
     <microservice.mainClass>org.wso2.carbon.mss.sample.Application</microservice.mainClass>  
     <appcloud.version>1.0.7</appcloud.version>  
   </properties>

[1] https://docs.wso2.com/display/AppCloud/Published+APIs

Oauth custom basic authenticator with WSO2 IS 5.1.0

2016-03-04T01:35:00.001-08:00

WSO2 Identity Server supports Oauth2 authorization code grant type with basic authentication OOTB. But basic authentication is done only with WSO2 user store. So there could be use cases that basic authentication has to be done against some other system. In this case you follow below steps to achieve your requirement.

First you need to create an class which extends AbstractApplicationAuthenticator and implements LocalApplicationAuthenticator. Because this class is going to act as your application authenticator so it needs to be an implementation of application authenticator interface and to achieve this it needs to be a local authenticator as well. [2]

 public class CustomBasicAuthenticator extends AbstractApplicationAuthenticator implements LocalApplicationAuthenticator {  

Then you need to override the initiateAuthenticationRequest method so you can redirect to the page to enter user and password. In my sample I redirected to the page that is used by our default basic authenticator[1]. Code goes as follows.

      @Override  
      protected void initiateAuthenticationRequest(HttpServletRequest request,  
                             HttpServletResponse response,  
                             AuthenticationContext context)  
                throws AuthenticationFailedException {  
           //TODO: Implement custom redirecting to a custom login page if needed.  
           String loginPage = ConfigurationFacade.getInstance().getAuthenticationEndpointURL();  
           String queryParams =  
                     FrameworkUtils.getQueryStringWithFrameworkContextId(context.getQueryParams(),  
                                               context.getCallerSessionKey(),  
                                               context.getContextIdentifier());  
           try {  
                String retryParam = "";  
                if (context.isRetrying()) {  
                     retryParam = "&authFailure=true&authFailureMsg=login.fail.message";  
                }  
                response.sendRedirect(response.encodeRedirectURL(loginPage + ("?" + queryParams)) +  
                           "&authenticators=BasicAuthenticator:" + "LOCAL" + retryParam);  
           } catch (IOException e) {  
                throw new AuthenticationFailedException(e.getMessage(), e);  
           }  
      }

Then you need to override processAuthenticationResponse method so you can do your own authentication. In my sample I didn’t do any authentication. You have to implement it to call your rest api. Then in your requirement you mentioned that you need to provision user to IS. So the code segment inside the if condition is to do that. In my sample I provisioned the user to the super tenant primary user store. You can decide on where to add user to.

      @Override  
      protected void processAuthenticationResponse(HttpServletRequest httpServletRequest,  
                                  HttpServletResponse httpServletResponse,  
                                  AuthenticationContext authenticationContext)  
                throws AuthenticationFailedException {  
           String username = httpServletRequest.getParameter(CustomBasicAuthenticatorConstants.USER_NAME);  
           String password = httpServletRequest.getParameter(CustomBasicAuthenticatorConstants.PASSWORD);  
           boolean isAuthenticated;  
           //TODO: Call the rest api to validate username and password  
           isAuthenticated = true;  
           //TODO: Here this is provisioned to the primary user store under super tenant. This can be changed to provision to the correct place.  
           if(isAuthenticated) {  
                UserStoreManager manager;  
                try {  
                     manager = CustomBasicAuthenticatorServiceComponent.getRealmService().getTenantUserRealm(-1234)  
                                             .getUserStoreManager();  
                } catch (UserStoreException e) {  
                     String msg = "Error while retrieving the user realm";  
                     LOGGER.error(msg, e);  
                     throw new AuthenticationFailedException(msg, e);  
                }  
                try {  
                     if (!manager.isExistingUser(username)) {  
                          manager.addUser(username, password, new String[0], new HashMap<String, String>(), null, false);  
                     }  
                } catch (UserStoreException e) {  
                     String msg = "Error while provisioning the user";  
                     LOGGER.error(msg, e);  
                     throw new AuthenticationFailedException(msg, e);  
                }  
                authenticationContext.setSubject(AuthenticatedUser.createLocalAuthenticatedUserFromSubjectIdentifier(username));  
           }  
      }

And you need to implement the can handle method of the authenticator. In the sample I used the implementation same as basic authenticator.

      @Override  
      public boolean canHandle(HttpServletRequest httpServletRequest) {  
           String userName = httpServletRequest.getParameter(BasicAuthenticatorConstants.USER_NAME);  
           String password = httpServletRequest.getParameter(BasicAuthenticatorConstants.PASSWORD);  
           if (userName != null && password != null) {  
                return true;  
           }  
           return false;  
      }

Then you need to make this bundle an osgi bundle and register the authenticator in osgi context. That’s done by service component class.

 /*  
  * Copyright (c) WSO2 Inc. (http://www.wso2.org) All Rights Reserved.  
  *   
  * WSO2 Inc. licenses this file to you under the Apache License,  
  * Version 2.0 (the "License"); you may not use this file except  
  * in compliance with the License.  
  * You may obtain a copy of the License at  
  *   
  * http://www.apache.org/licenses/LICENSE-2.0  
  *   
  * Unless required by applicable law or agreed to in writing,  
  * software distributed under the License is distributed on an  
  * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY  
  * KIND, either express or implied. See the License for the  
  * specific language governing permissions and limitations  
  * under the License.  
  */  
 package org.wso2.identity.application.authenticator.custom.basicauth.internal;  
 import org.wso2.identity.application.authenticator.custom.basicauth.CustomBasicAuthenticator;  
 import org.apache.commons.logging.Log;  
 import org.apache.commons.logging.LogFactory;  
 import org.osgi.service.component.ComponentContext;  
 import org.wso2.carbon.identity.application.authentication.framework.ApplicationAuthenticator;  
 import org.wso2.carbon.user.core.service.RealmService;  
 import java.util.Hashtable;  
 /**  
  * @scr.component name="application.authenticator.dbevaldev.component" immediate="true"  
  * @scr.reference name="realm.service"  
  * interface="org.wso2.carbon.user.core.service.RealmService"cardinality="1..1"  
  * policy="dynamic" bind="setRealmService" unbind="unsetRealmService"  
  */  
 public class CustomBasicAuthenticatorServiceComponent {  
   private static final Log LOGGER = LogFactory.getLog(CustomBasicAuthenticatorServiceComponent.class);  
      private static RealmService realmService;  
   protected void activate(ComponentContext context) {  
     try {  
       CustomBasicAuthenticator dbevaldevauthenticator = new CustomBasicAuthenticator();  
       Hashtable<String, String> props = new Hashtable<String, String>();  
       context.getBundleContext().registerService(ApplicationAuthenticator.class.getName(), dbevaldevauthenticator, props);  
       if (LOGGER.isDebugEnabled()) {  
         LOGGER.debug("Custom authenticator bundle is activated");  
       }  
     } catch (Exception e) {  
       LOGGER.fatal(" Error while activating custom authenticator ", e);  
     }  
   }  
   protected void deactivate(ComponentContext context) {  
     if (LOGGER.isDebugEnabled()) {  
       LOGGER.debug("Custom authenticator bundle is deactivated");  
     }  
   }  
      public static RealmService getRealmService() {  
           return realmService;  
      }  
      protected void setRealmService(RealmService realmService) {  
           if(LOGGER.isDebugEnabled()) {  
                LOGGER.debug("Setting the Realm Service");  
           }  
           CustomBasicAuthenticatorServiceComponent.realmService = realmService;  
      }  
      protected void unsetRealmService(RealmService realmService) {  
           if(LOGGER.isDebugEnabled()) {  
                LOGGER.debug("UnSetting the Realm Service");  
           }  
           CustomBasicAuthenticatorServiceComponent.realmService = null;  
      }  
 }

Then you can build this with maven and copy the jar file to the following folder

$IS_HOME/repository/components/dropins

Then restart the Identity Server and browse to the carbon console. Then from the left menu Select Identity > Service Providers > List.

There you can select the service provider for the application and click on edit link to edit the particular service provider and expand the section Local & Outbound Authentication Configuration and there select Local authentication and select the newly added authenticator. And then select update.

Now you can trigger the authorization code grant type by invoking oauth2/authorize endpoint in IS. You can download the sample code from [3].

[1] https://github.com/wso2/carbon-identity/blob/master/components/application-authenticators/org.wso2.carbon.identity.application.authenticator.basicauth/src/main/java/org/wso2/carbon/identity/application/authenticator/basicauth/BasicAuthenticator.java

[2] https://docs.wso2.com/display/IS500/Creating+Custom+Authenticators
[3] https://drive.google.com/file/d/0ByTCb2KmTk76WHluNjFHcEVtSzg/view?usp=sharing

Adding a servlet to Orion (Eclipse Cloud IDE) from a package built using maven.

2016-02-22T23:05:00.002-08:00

Orion is an open source project and it is under the Eclipse Cloud Development top-level project. It’s an cloud IDE which is available online and also which can be hosted in on premise. [1] [2]. Orion can work with git repos, create git repos or work with local code same as Eclipse.

Orion has two repos. For the backend server code [2] and for the client / ui code [3]. Orion server is an osgi server based on eclipse equinox. But the problems are they have not implemented the dropins concept. Because of that adding an external bundle to Orion Server is not straightforward. We have to edit the bundles.info to add a bundle to Orion server. And Orion is using jetty for the servlets.

As mentioned Orion is working with Jetty and to add a servlet via a bundle is easy if we use eclipse itself to develop and bundle the package. But if we use maven to build it then the problems arises. What we have to do is add an entry to plugins.xml and package the plugins.xml to the bundle we are creating.

So to do this what has to be done is add a plugins.xml (You can copy an existing plugins.xml from orion code base) to the resources directory and edit it to have your servlets in it. And in the pom.xml under the build tag add following section. You can look into the sample code [4].

     <resources>  
       <resource>  
         <directory>src/main/resources</directory>  
         <filtering>true</filtering>  
         <includes>  
           <include>build.properties</include>  
           <include>bundle.properties</include>  
           <include>plugin.xml</include>  
         </includes>  
       </resource>  
     </resources>

Sample plugins.xml is as below.

 <?xml version="1.0" encoding="UTF-8"?>  
 <?eclipse version="3.4"?>  
 <plugin>  
   <extension  
      point="org.eclipse.equinox.http.registry.servlets">  
    <servlet  
       alias="/helloworld"  
       class="org.orion.sample.servlet.servlet.HelloWorldServlet">  
    </servlet>  
    <serviceSelector  
       filter="(other.info=org.eclipse.orion)">  
    </serviceSelector>  
   </extension>  
 </plugin>

Servlet code will be the following

 package org.orion.sample.servlet.servlet;  
 import org.eclipse.orion.server.servlets.OrionServlet;  
 import javax.servlet.ServletException;  
 import javax.servlet.http.HttpServletRequest;  
 import javax.servlet.http.HttpServletResponse;  
 import java.io.IOException;  
 public class HelloWorldServlet extends OrionServlet {  
      @Override  
      protected void service(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {  
           resp.getWriter().println("Hello JCG, Hello OSGi");  
      }  
      public HelloWorldServlet() {  
           super();  
      }  
 }

Since to deploy the bundle to Orion it has to be an osgi bundle you have add maven scr and maven bundle plugins to your project as well. So the build tag in the pom.xml should be like below.

   <build>  
     <resources>  
       <resource>  
         <directory>src/main/resources</directory>  
         <filtering>true</filtering>  
         <includes>  
           <include>build.properties</include>  
           <include>bundle.properties</include>  
           <include>plugin.xml</include>  
         </includes>  
       </resource>  
     </resources>  
     <plugins>  
       <plugin>  
         <groupId>org.apache.felix</groupId>  
         <artifactId>maven-scr-plugin</artifactId>  
         <version>1.7.2</version>  
         <executions>  
           <execution>  
             <id>generate-scr-scrdescriptor</id>  
             <goals>  
               <goal>scr</goal>  
             </goals>  
           </execution>  
         </executions>  
       </plugin>  
       <plugin>  
         <groupId>org.apache.felix</groupId>  
         <artifactId>maven-bundle-plugin</artifactId>  
         <version>2.3.5</version>  
         <extensions>true</extensions>  
         <configuration>  
           <instructions>  
             <Bundle-SymbolicName>${project.artifactId};singleton:=true</Bundle-SymbolicName>  
             <Bundle-Name>${project.artifactId}</Bundle-Name>  
             <Export-Package>  
               org.orion.sample.servlet.*,  
             </Export-Package>  
             <DynamicImport-Package>*</DynamicImport-Package>  
             <Require-Bundle>  
               org.eclipse.jetty.server,  
               org.eclipse.jetty.servlet,  
               org.eclipse.jetty.http,  
               org.eclipse.jetty.io,  
               org.eclipse.jetty.security,  
               org.eclipse.jetty.util,  
               org.eclipse.orion.server.core,  
               org.eclipse.orion.server.git,  
               org.eclipse.orion.server.servlets,  
               org.eclipse.equinox.http.registry  
             </Require-Bundle>  
           </instructions>  
         </configuration>  
       </plugin>  
     </plugins>  
   </build>

And you have to add the Service Component class to get it activated at the startup. Code would be like following.

 package org.orion.sample.servlet.internal;  
 import org.osgi.service.component.ComponentContext;  
 /**  
  * @scr.component name="org.orion.sample.servlet.internal.ServletSampleServiceComponent" immediate="true"  
  */  
 public class ServletSampleServiceComponent {  
      protected void activate(ComponentContext context) {  
           System.out.println("Registration Complete");  
      }  
 }

Now you can run mvn clean install on the package and build the jar file. Now go in to the Orion server folder and create a folder named dropins and copy your jar file in to dropins. Now edit

$ORION_HOME/configuration/org.eclipse.equinox.simpleconfigurator/bundles.info

file and following in the end add following line.

org.orion.sample.servlet,1.0.0,dropins/org.orion.sample.servlet-1.0.0.jar,4,false

[1] https://wiki.eclipse.org/Orion
[2] https://github.com/eclipse/orion.server
[3] https://github.com/eclipse/orion.client
[4] https://drive.google.com/open?id=0ByTCb2KmTk76OVMtcnplQnpOajQ

WSO2 AppFactory - Using ESB Apptype

2015-07-29T03:27:00.001-07:00

In next version of WSO2 AppFactory (2.2.0), it's going to introduce a new apptype ESB apptype. With this apptype, it's going allow users to develop WSO2 ESB Capps (CAR) with WSO2 AppFactory. In this article I will give you a guide line to follow when you are developing a Capp using WSO2 AppFactory.

WSO2 AppFactory ESB apptype is the first multi module apptype supported by it. It will contain 4 modules in the sample project. They are as below.

Resources module
Resources CAR module
Synapse Config (Proxy Service) module
Main Car module

Development

There are several rules that developer should obey when developing an ESB type application. Which are as follows.

Developers can add any number of modules to the project. But always there should only be two CAR modules. Which should be resources CAR and main CAR which will contain all synapse configs.
All synapse config names should contain the version number. Like foosequence-1.0.0. And the synapse config file name also should contain the version number in the same manner.
All modules should start with the application ID. This rule is not to conflict artifacts in between applications. If two applications contained artifacts in same name they could conflict.
Main Car module artifact id should be similar to the application ID.

It is recommended to use WSO2 Developer Studio to develop WSO2 AppFactory ESB type application. Developer Studio will validate the project structure and it will help developer to follow above rules. If some one is going to edit it by some other method, before accepting the commit it will check whether the required structure is there and will decide to accept it or reject it.

LifeCycle Management and Resources Management

When the ESB application is promoted it will still keep using the development endpoints / resources which is mentioned in the Resources CAR module. So the users in next stage (QA / DevOps) will need to update this resources CAR. So there will be an UI to upload a resources CAR for ESB applications. QAs and DevOps will need to checkout the code from the source code location which is mentioned in the Application Home page and they will need to edit registry resources and endpoints to match their endpoints and then they can build it and upload there Resources CAR so it will get deployed and main car will change it's endpoints to new ones.

[1] http://wdfdo1986.blogspot.com/2015/07/wso2-app-factory-life-cycle-of.html

[2] http://wdfdo1986.blogspot.com/2015/07/wso2-app-factory-developing-new.html

WSO2 App Factory - Developing New Application Type

2015-07-24T04:42:00.003-07:00

This post will describe how to use extension points to develop an application type. In [1] you can find step by step guide of adding a new application type and its runtime. In this post I will explain following.

Adding new Application Event Handlers
Maven Archetype creation for App Factory Applications
Extending the Application Type Processor Interface
Extending Deployer Interface

Adding new Application Event Handlers

You can find the interface ApplicationEventsHandler in [3] Lets say you need to do some special stuff for your application type (like create a database when creating an application to support your app type), you can extend methods of this to do that. For example we are creating a datasource using an handler when we create a Data Service type application. You can find that code in [4].

After you develop your handler you can add it to an OSGI bundle and then deploy it to the AppFactory server's following location.

$APPFACTORY_HOME/repository/components/dropins

And then you can edit the order of your handler by mentioning the necessary priority in the appfactory configuration which can be found in the following location.

$APPFACTORY_HOME/repository/conf/appfactory/appfactoy.xml

In this configuration file you can find a tag named EventHandlers. You can insert a new tag to this place with the name of your handler class name and set the desired priority value as it is done for other handlers.

Figure 1 : Event Handlers Priority Settings

Maven Archetype creation for App Factory Applications

In WSO2 App Factory when we create an application we generate a sample code for that application. This is done using maven archetypes. We have created a maven archetype for all apptypes. Maven archetypes can be created for non maven applications as well. For an example even for .NET app type we are having a maven archetype even though its built using MS Build.

[5] describes how to create a maven archetype. And you can also take some hints from my previous post [7] for maven archetype creation. When we create an archetype for WSO2 AppFactory we have to add few more resources in to it. That is for the initial deployment. This initial deployment phase was introduced to reduce the time spent to create an application. So what happens there is we include built artifacts in to the maven archetype and we bundle them using assembly plugin and deploy the created artifact[6].

Figure 2 shows the tree structure of web application archetype. I use this to explain since this is the simplest and common structure. Here archetype-resources folder contains the content of the archetype. Apart from the src directory you can see there is a directory named built_artifact. This is the place that we use to keep built artifacts that we are going to keep to create the initial artifact using maven assembly plugin. And the assembly.xml and the bin.xml placed at the same level will be used to run maven assembly plugin and create the initial artifact.

Figure 2 : Tree structure of Web app archetype

You can see the assembly.xml and bin.xml here. When we trigger follwing command on this folder it will create an .war file with the content of the built_artifact folder and will copy it to a folder named ${artifactId}_deploy_artifact which is located one level outside the application code. And in the initial deployment phase it will pick this artifact to deploy.

mvn clean install -f assembly.xml

Figure 3 : bin.xml

Figure 4 : assembly.xml

Figure 5 is the structure of maven archetype for ESB apptype which is the most complex apptype created in WSO2 AppFactory up to now. Here you can see that there is no assembly.xml and bin.xml and instead of that there is a pom.xml inside built_artifact folder and bin.xml is also there in the same level. So we have run following command inside the built_artifact folder and how this extension can be done will be described in later in this article.

mvn clean install

Figure 5 : Tree structure of ESB maven archetype

Extending the Application Type Processor Interface

Application Type Processor interface can be found at [8]. AbstractApplicationTypeProcessor is an abstract implementation of the interface which is recommended to use[9]. But there could be cases where that it needs to be implemented from the beginning. This processor is triggered in several places which will do something specific to an application type.

When we create an application when the repository get initiated[2], generateApplicationSkeleton method will get invoked. In [9] it is calling a method initialDeployArtifactGeneration which is not mentioned in the interface but anyone extending [9] can override that. The behavior I mentioned in the earlier section about ESB apptype initial artifact generation.

Extending Deployer Interface

The Deployer Interface [10] will be used to deploy the artifacts to the PaaS Artifact repository. Underlying PaaS will be the responsible one to copy it to the actual running server instance. If you need to do something special with your apptype you can use this interface. Anyway again the recommended class to extend is AbstractDeployer [11]. Most common use of this will be changing the artifacts that you choose to deploy. So you can override getArtifact method in [11] so it will select the artifacts to deploy in your algorithm.

And the next thing would be Initial Artifact Deployment. As I explained earlier WSO2 AppFactory will deploy an initial artifact generated directly from the archetype generated code. So this is usually done using the class InitialArtifactDeployer [12]. But if you want to change some behavior of this class then again you can extend this class [12] and add it to WSO2 AppFactory and mention it in apptype (apptype.xml) under a parameter name "InitialDeployerClassName".

References

[1] Adding a new App Type and its Runtime
[2] WSO2 Appfactory LifeCycle Of
[3] ApplicationEventsHandler.java (org.wso2.carbon.appfactory.core.ApplicationEventsHandler)
[4] DSApplicationListener.java
[5] Guide Creating Archetypes
[6] Maven Assembly Plugin
[7] How to include artifactid in folder name or content of a file
[8] ApplicationTypeProcessor.java (org.wso2.carbon.appfactory.core.apptype.ApplicationTypeProcessor)
[9] AbstractApplicationTypeProcessor.java
[10] Deployer.java (org.wso2.carbon.appfactory.core.Deployer)
[11] AbstractDeployer.java (org.wso2.carbon.appfactory.deployers.AbstractDeployer)
[12] InitialArtifactDeployer.java (org.wso2.carbon.appfactory.deployers.InitialArtifactDeployer)

WSO2 App Factory - Life Cycle of an application

2015-07-23T04:27:00.002-07:00

This post will try to explain the general life-cycle of an application in WSO2 AppFactory. WSO2 AppFactory is a place where multiple project teams can collaboratively create run and manage enterprise applications[1]. With WSO2 AppFactory users can create an complex application and push it to production within a matter of couple of hours.

Figure 1 - App Cloud Create Application Page

If you go to WSO2 AppFactory and create an application it will do several operations that generates the resources needed for your application such as

Create Repository
Generate Sample Code for the Application
Create Build Job
Create Issue Tracker Project
Deploy initial artifact in to PAAS artifact repository
etc..

When you hit create application it will create an instance of the application RXT installed in WSO2 AppFactory and then will call a bpel which is hosted in the BPS. Any one can edit this bpel to add their workflow in to it. Currently this will just trigger the AF service to trigger on creation event of Application Event Handlers. There are set of Application Event Handlers registered in AF. You can develop a new Application Event Handler class and add it to an OSGI bundle and copy to following location and start AF and that new handler will get invoked.

$APPFACTORY_HOME/repository/components/dropins

And you can configure the order of the handlers by setting the priority of the handler by setting priority in the AppFactory configuration placed in following location.

$APPFACTORY_HOME/repository/conf/appfactory/appfactoy.xml

I will explain this in detail in my next post [2].

These handlers will be responsible for the operations mentioned above. Figure 2 is about the flow of this Application Creation process.

Figure 2 : Create Application Flow

When the application is created trunk version will get created in the repository. Next step would be to create a new version from the trunk version to promote it to next stages.

Figure 3 : Repositories and Builds page

When you click the create branch it will trigger same kind of flow and will create the new version and new build jobs.

Figure 4 : Flow of creating a new version in an Application

Once you create a new version then you can promote this version to next stages (Development -> Testing -> Production). When you promote it will deploy the artifacts to next stages.

[1] http://wso2.com/cloud/app-factory/
[2] http://wdfdo1986.blogspot.com/2015/07/wso2-app-factory-developing-new.html

WSO2 Products - How User Stores work

2015-07-09T05:19:00.001-07:00

Today we keep our users and profiles in several forms. Some times they are in a LDAP. Some uses Active Directory (AD). Some uses databases and etc. WSO2 Products are written in a way any of these format could support. If some one have their own way of storing users and any one can easily plug them in to WSO2 products by just writing a custom user store. In this post I will explain how these user stores works and the other components connected to them.

When we discuss about user management in WSO2 world, there are several key components. They are

User Store Manager
Authorization Manger
Tenant Manager

In simple user management we need to authorize some user to some action / permission. Normally we group these actions / permissions as groups and assign these groups / roles to users. So there are two kind of mappings that we need to consider. They are

User to Role Mapping
Role to Permission Mapping

User to Role Mapping is managed by user store implementation and Role to Permission mapping is managed by authorization manger implementation. These things are configured in the configuration file under [1].

Tenant Manager comes in to play when Multi Tenancy is considered. This is configured under [2]. Lets discuss this later.

User Management

By default WSO2 products (except WSO2 Identity Server) it stores every thing in DB. There it use [3] as the user store manager implementation. In WSO2 Identity Server, it's shipped with an internal LDAP and users are stored in that LDAP. So there it uses the [4] as the implementation of the User Store Manager. And by default all WSO2 servers uses the DB to store Role to Permission Mapping. There it uses [5] as the authorization manager implementation.

I will explain the WSO2 Identity server case since it contains most of the elements. Since it has an LDAP user store, all users in the system, all roles in the system and the all the user to role mappings are saved in the LDAP. User Store Manager Implementation which is based on the interface [6]. WSO2 Products contains an Abstract User Store Implementation which includes the most of the common implementation is done and extension points provided for plug external implementations [7]. It is recommended to use [7] as the base when writing a user store manager always. All users, roles and user role mappings are managed through [4] which are in LDAP. And all the Role to Permissions mappings are persisted in DB and it was handled via [5].

Figure 1 : User Stores and Permission Stores

Figure 1 is about the relationships among users, roles and permissions and where they are stored and who is handling them.

Multi tenancy

Lets get in to multi tenancy now. Some of you may already know what is multi tenancy. But on behalf of the others in multi tenancy we create a space ( a tenant ) which is isolated from everything and nobody other than the people in the tenant don't know the existence of the tenant.

So in WSO2 products OOTB its supporting for a completely isolated tenants. Each tenant will have their own artifact space, registry space and user store (We call this user realm). Figure 2 graphically explains this story.

Figure 2 : Each Tenant having their own user store

Since there could be lots of tenants in the system in WSO2 products, it won't load all the tenants to the memory. It will load the tenant to the memory when the tenant is active. And when the tenant is idle for some time it will get unloaded. When tenant get loaded it will load registry, user realm and artifacts belongs to the tenant.

In this user realm it contains all the users, roles, permissions and their mappings. Realm gets loaded from the implementation of [8] which we mention in [2]. So by having your own implantation you can plug your tenant structure to WSO2 Products.

In next post I will explain plugging a custom LDAP structure to WSO2 Products.

[1] $CARBON_HOME/repository/conf/user-mgt.xml
[2] $CARBON_HOME/repository/conf/tenant-mgt.xml
[3] org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager
[4] org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager
[5] org.wso2.carbon.user.core.authorization.JDBCAuthorizationManager
[6] UserStoreManager.java
[7] AbstractUserStoreManager.java
[8] org.wso2.carbon.user.core.config.multitenancy.MultiTenantRealmConfigBuilder
[9] MultiTenantRealmConfigBuilder.java

How to include artifactid in a folder or file name or content of a file in a maven archetype.

2015-07-09T03:58:00.003-07:00

When we create maven archetypes, specially the multi module ones, we might need to include the artifact id in to a folder or file name or include it to a content of a file. This can be done very easily.

To include artifactid to a folder or file name you just have to add the place holder __rootArtifactId__.

**Note that there are two '_' characters before and after the word rootArtifactId.

So for example if you want a file name like -development.xml, then you can simply name it as __rootArtifactId__-development.xml. When file name is mentioned like this inside the archetype, when you run archetype generate command it will replace with the artifactid provided.

Next thing is how to include this inside a file. This was tricky. I couldn't find a way to do it first. So I keep trying things and it works and it's simple. You can simply add the place holder ${rootArtifactId}.

In the same pattern you can use other parameters like version as well by using __version__ and ${version} respectively.

Enjoy !!!!

Tips to write an Enterprise Application On WSO2 Platform

2014-06-24T02:50:00.000-07:00

Enterprise applications or Business Applications, are complex, scalable and distributed. They could deploy on corporate networks, Intranet or Internet. Usually they are data centric and user-friendly. And they must meet certain security, administration and maintenance requirements.

Typically Enterprise Applications are large. Which is multi user, runs on clustered environments, contains large number of components, manipulates large amount of data and may use parallel processing and distributed resources. And they will try to meet some business requirements and at the same time it should provide robust and maintenance, monitoring and administration.

Here are some features and attributes that may include in an Enterprise Application.

Complex business logic.
Read / Write data to / from databases.
Distributed Computing.
Message Oriented Middleware.
Directory and naming services
Security
User Interfaces (Web and / or Desktop)
Integration of Other systems
Administration and Maintenance
High availability
High integrity
High mean time between failure
Do not lose or corrupt data in failures.

The advantages of using WSO2 platform to develop and deploy an Enterprise Application is that most of above are supported by WSO2 platform itself. So in this blog entry I am going to provide some tips to develop and deploy an Enterprise Application in WSO2 platform.

Read / Write data to / from databases.

In WSO2 platform convention of using databases is access them through datasources. Here the developer can use WSO2 SS (Storage Server) [1] to create the databases. [2]. So the developer of the Application can create the database needed and if needed add the data to database console provided by WSO2 SS which is explained in [2]. For security reasons we can restrict developers to use the mysql instances only through WSO2 SS by restricting the access outside the network.

After creating a database next step would be to create a datasource. For this purpose developer can create a datasource in WSO2 AS (Application Server) [3] and [4] explains how to add and manage data sources. As it is explained in [5] developer can expose the created data source as a JNDI resource and developer can use the data source/s in the application code as explained there.

Use Distributed Computing and Message Oriented Middleware provided by WSO2

WSO2 ESB can be used to add Distributed computing to the application. [6] and [7] explains how the developer can use WSO2 ESB functionalities to add Distributed Computing to the his / her application.

WSO2 ESB also supports JMS (Java Messaging Service) [8] which is a widely used API in Java-based Message Oriented Middleware. It facilitates loosely coupled, reliable, and asynchronous communication between different components of a distributed application.

Store Configuration, Endpoints in WSO2 Registry.

And developer can store configuration, endpoints in registry provided by each WSO2 product. So registry have three parts.

Governance - Shared across the whole platform
Config - Shared across the current cluster
Local - Only available to current instance

Normally developer need to store data in governance if that data needs to be accessed by other WSO2 products as well. Otherwise he/she needs to store data in config registry.

Directory And Naming Services Provided by WSO2 Platform

All WSO2 Products can be use with LDAP, AD or any other Directory or Naming services and WSO2 Carbon APIs provide developer the APIs which can do operations with these Directory or Naming services. This is handled using User Store Managers implemented in WSO2 products [9]. Anyone who will use WSO2 products can extend these User Store Managers to map it to their Directory structure. [10] provides a sample of how to use these Carbon APIs in side application to access the Directory Services from the Application.

Exposing APIs and Services

Web app developer can expose some APIs / Webservices from his / her application and he / she can publish them via WSO2 API Manager [21] so everyone can access them. In this way the Application can be integrated in to the other systems and the application can use the existing APIs without implementing them again.

And there is another commonly used feature in WSO2 Platform. The data sources created using WSO2 AS / WSO2 DSS can be exposed as data services and these data services can be exposed as APIs from WSO2 API Manager [22] .

The advantage of using WSO2 API Manager in this case is mainly security. WSO2 API Manager provides oauth 2.0 based security.

Security

When providing security we can provide security to the application by providing authentication and authorization. And we can provide security to the deployment by applying Java security and Secure vaults. And services deployed can be secured using Apache Rampart [11] [12].

To provide authentication and authorization to the Application developer can use the functionalities provided by the WSO2 IS (Identity Server) [13]. Commonly SAML SSO is used to provide authentication. [14] explains how SSO works, how to configure to work with SAML SSO and so on.

For authorization purposes developer can use the Carbon APIs provided in WSO2 products which is described in [15].

Java Security Manager can be used with WSO2 products so the deployment can be secured with the security provided by the policy file. As explained in [16] Carbon Secure Vaults can be used to store passwords in a secure way.

Develop an Application to deploy on Application Server

[20] provides a user guide to develop and deploy an java application on WSO2 AS. This documentation discuss about class loading, session replication, writing Java, JAX - RS, JAX - WS, Jaggery and Spring applications, Service Development Deployment and Management, usage of JAVA EE and so on.

Administration, Maintenance and Monitoring

WSO2 BAM (Business Activity Monitor) [17] can be use to collect logs and create some dashboards which will let people to monitor the status of the system. [18] explains how data can be aggregated, processed and presented with WSO2 BAM.

Clustering

WSO2 Products which are based on Apache Axis2, Can be clustered. [19] provides clustering tips and how to cluster WSO2 products. By clustering the high availability can be achieved in the system.

References

[1] http://wso2.com/products/storage-server/

[2] https://docs.wso2.org/display/SS110/Creating+Databases

[3] http://wso2.com/products/application-server/

[4] https://docs.wso2.org/display/AS521/Datasources

[5] https://docs.wso2.org/display/AS521/Exposing+Datasources+as+JNDI+Resources

[6] https://docs.wso2.org/display/ESB460/Distributed+Computing+Evolution

[7] https://docs.wso2.org/display/IntegrationPatterns/Enterprise+Integration+Patterns+with+WSO2+ESB

[8] https://docs.wso2.org/display/ESB480/Java+Message+Service+(JMS)+Support

[9] https://docs.wso2.org/display/IS460/Configuring+Primary+User+Stores

[10] https://docs.wso2.org/display/AF200/Consuming+User+Management+APIs+from+Applications

[11] http://axis.apache.org/axis2/java/rampart/

[12] http://wso2.com/taxonomy/term/267/all

[13] http://wso2.com/products/identity-server/

[14] https://docs.wso2.org/display/IS460/Managing+Single+Sign-On

[15] https://docs.wso2.org/pages/viewpage.action?pageId=32352001

[16] https://docs.wso2.org/display/Carbon420/WSO2+Carbon+Secure+Vault

[17] http://wso2.com/products/business-activity-monitor/

[18] https://docs.wso2.org/display/BAM241/User+Guide

[19] https://docs.wso2.org/display/CLUSTER420/WSO2+Clustering+and+Deployment+Guide

[20] https://docs.wso2.org/display/AS521/User+Guide

[21] http://wso2.com/products/api-manager/

[22] https://docs.wso2.org/display/AS521/Data+Services

Dell Vostro 1520 Boot problem, Recovering & restore Grub from a live Ubuntu cd.

2011-02-25T09:30:00.000-08:00

In Vostro 1520 Dell Laptop series, when Ubuntu installed with windows (Dual Boot), due to a problem in bios the MBR will get corrupted and after that you will be unable to see the Grub loader at the startup and you want be able to load either windows or Ubuntu.

This happens only because of the corruption at MBR record or partition table.

So by reinstalling grub to the partition u can resolve this problem. The steps of resolving are given below.

First you have to boot from the Ubuntu Live CD

Then you have to find the sd number of the partition that grub is installed in. ( If you have a separate /boot partition it should be the boot partition sd number otherwise it should be the Ubuntu file system sd number. ). And here I assume that sd number is x.

Open the Terminal and execute following codes

sudo mkdir /mnt/root -- Setup a folder before change root.

Using this code Mount the partition in to the folder created.

sudo mount -t ext3 /dev/sdax /mnt/root

Mount and bind necessary places using following code

Run this if /mnt/root/proc and /mnt/root/dev doesn't exist

sudo mkdir /mnt/root/proc
sudo mkdir /mnt/root/dev

sudo mount -t proc none /mnt/root/proc

sudo mount -o bind /dev /mnt/root/dev

Now change the root to the configured folder using this code

sudo chroot /mnt/root /bin/bash

Install the grub to specified device.

grub-install /dev/sdax

Now you are done. Restart and check.

You will see that the grub is loading. Good luck !!!

Linux Mint updates bug found

2010-10-06T04:02:00.000-07:00

Bug Description : Failed to fetch

http://packages.linuxmint.com/dists/isadora/Release   (malformed Release  file?) Unable to find expected entry  multiverse/binary-i386/Packages in  Meta-index file Some index files failed to download, they have been  ignored, or old ones used instead.

Reasoning and Solution : There is no multiverse section in the Mint repository... please edit your /etc/apt/sources.list and remove the "multiverse" keyword from the lines referring to the Mint repositories. 

How to Fix it:

 This command will open the sources list. 

~$ sudo gedit /etc/apt/sources.list

This is my file before editing and after editing

Before :-

deb http://packages.linuxmint.com/ isadora main upstream import multiverse import

deb http://archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse

deb http://archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu/ lucid-security main restricted universe multiverse

deb http://archive.canonical.com/ubuntu/ lucid partner

deb http://packages.medibuntu.org/ lucid free non-free

# deb http://archive.getdeb.net/ubuntu lucid-getdeb apps

# deb http://archive.getdeb.net/ubuntu lucid-getdeb games

After:-

deb http://packages.linuxmint.com/ isadora main upstream import

deb http://archive.ubuntu.com/ubuntu/ lucid main restricted universe

deb http://archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe

deb http://security.ubuntu.com/ubuntu/ lucid-security main restricted universe

deb http://archive.canonical.com/ubuntu/ lucid partner

deb http://packages.medibuntu.org/ lucid free non-free

# deb http://archive.getdeb.net/ubuntu lucid-getdeb apps

# deb http://archive.getdeb.net/ubuntu lucid-getdeb games

How to fix grub loader when its not loading at the boot

2010-07-20T22:32:00.000-07:00

When you installed or restore a Microsoft Windows after the Linux installation you won't see the grub loader is loading at the boot.

This is not a error of grub or Linux. But you couldn't load Linux at this situation.

There are few ways of fixing this issue. One way is using super grub disk.

Here I will tell you an another way of fixing this issue.

You just have to boot using any type of Linux CD/DVD

Now Open the terminal and type

sudo grub

and type the password to enter the kernel mode.

Now you have to find out few things.

First find the hd number of the hard disk.

This is normally zero since we are using one hard disk only.

But if we are using more of them we have to find out the hd number of the hard disk.

Then find the sd number of the partition which contains grub.

If you are not partitioned a separate /boot partition at the Linux installation this is same as the Linux partition.

If you are using Debian version you can start the application called Partition Editor and find out these things.

Here I will take hd number is x and sd number is y.

After typing sudo grub and enter the password you will be switched in to new prompt ( grub prompt ) like below

grub>

Now Enter

grub> root (hdx,y)

Now Restart the machine and you will find the grub loader is loading.

Enjoy!!!

Customize your grub loader by just editing a file

2010-07-20T22:10:00.000-07:00

You can find a file called menu.lst in the path /boot/grub. By editing this file you can customize your Grub loader easily. 

First we can set the time out amount of the grub. Go to Section ## timeout sec and set timeout value to an amount of seconds as you like.

Then you can Customize the view by changing the colors. It is under # Pretty colors.

Then comes the most important thing of the Grub menu list. Find out the kernel list in the File. Normally it is in the bottom. Sample section for one kernel is given below.

title        Linux Mint 7 Gloria, kernel 2.6.28-16-generic

root       faaab6f9-539b-4f1a-82fc-b5a18887d28d

kernel    /vmlinuz-2.6.28-16-generic root=UUID=6a2a464d-bf70-4b4a-8c8e-8dad3ccafe2c ro quiet splash 

initrd      /initrd.img-2.6.28-16-generic

quiet

You can always change the title as you wish. root is the UUID of the boot partition. If u don't have a separate boot partition then it is same as the partition containing Linux kernel. Then the generalize form is given below.

title        <Title of the kernel / os>

root      <UUID of the boot partition>

kernel    /vmlinuz-2.6.28-16-generic root=UUID= ro quiet splash 

initrd      /initrd.img-2.6.28-16-generic

quiet

Other details are about the Kernel versions. So don't try to edit them until its really needed and u exactly know what is doing. 

After finish the editing reboot and see what happened.

Find the menu list in the latest Grub Version

2010-07-18T21:59:00.000-07:00

I have noted that in the latest version of the Grub the file u have to edit to customize the Grub view or to Fix Errors had changed. It is no longer exists inside menu.lst file. Now it is in the grub.cfx file in the same location.

How to mount a partition permanently some where in the folder structure.

2010-02-15T02:22:00.000-08:00

First check this post and come to this section

How to edit file /etc/fstab

You needed to add the following line to the fstab file.

# / was on /dev/sdax during installation

UUID=xxxx  /media/xxxxxxxxx               xxx    relatime,errors=remount-ro a b

xxxx - UUID of the partition

x - number of the partition you needed to mount

xxx - file format of the partition

xxxxxxxxx  - Name of the Partition**

a - dump

b - pass

** Name of the partition should be the same as the Real name of the partition (The name we can see at the Computer (Simultaneous to My Computer in Microsoft Windows)).

An example line for this is

# /media/MyDisk was on /dev/sda8 during installation

UUID=64839f40-b2f1-412f-ae1a-c5a213ba449a /media/MyDisk  ntfs    relatime,defaults-ro 0 2

It is recommended to have name without spaces here. Because spaces can cause troubles.

How to overcome the problem Linux kernel image is not booting properly

2010-02-15T02:20:00.000-08:00

First check this post and come to this section

How to edit file /etc/fstab

This is the part of the fstab file that needed in this matter.

# / was on /dev/sdax during installation

UUID=xxxx /               ext3    relatime,errors=remount-ro a b

# swap was on /dev/sday during installation

UUID=yyyy none            swap    swc d

/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8               0       0

x should be the number of the partition of the ubuntu installed

y should be the number of swap

xxxx - UUID of Ubuntu partition

yyyy - UUID of swap

a - dump            (0 is my recommendation)

c - dump            (0 is my recommendation)      

b - pass            (1 is my recommendation)

d - pass             (0 is my recommendation)

How to edit file /etc/fstab

2010-02-15T02:06:00.000-08:00

This is how the my fstab file is looks like which is placed inside the etc folder in the Linux Folder structure.

You can overcome some problems by editing this file.

1. When your Linux image is not booting properly.

2. When you wan't to mount a partition permanently some where in the folder structure.

# /etc/fstab: static file system information.

#

# Use 'vol_id --uuid' to print the universally unique identifier for a

# device; this may be used with UUID= as a more robust way to name devices

# that works even if disks are added and removed. See fstab(5).

#

#

proc            /proc           proc    defaults        0       0

# / was on /dev/sda8 during installation

UUID=64839f40-b2f1-412f-ae1a-c5a213ba449a /               ext3    relatime,errors=remount-ro 0       1

# /home was on /dev/sda7 during installation

UUID=2ab422e9-b37d-4e7e-966f-6ca7d1d081cf /home               ext3    relatime,errors=remount-ro 0       2

# /boot was on /dev/sda6 during installation

UUID=faaab6f9-539b-4f1a-82fc-b5a18887d28d /boot               ext3    relatime,errors=remount-ro 0       3

# swap was on /dev/sda5 during installation

UUID=ab5f806d-8f4e-42b9-b67b-5618e9715585 none            swap    sw              0       0

/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8 0       0

Now we can study a line what it contains

# /home was on /dev/sda7 during installation

UUID=2ab422e9-b37d-4e7e-966f-6ca7d1d081cf /home               ext3    relatime,errors=remount-ro 0       2

This line contains following parts

# /home was on /dev/sda7 during installation
UUID=2ab422e9-b37d-4e7e-966f-6ca7d1d081cf
/home
ext3
relatime,errors=remount-ro 0 2

Now let me consider each one of these.

Actually I have separate home and boot partitions.

And when I installed linux for the second time I had to edit this file to use them as the actual home and boot partitions.

Part 1

This says that where the partition was in at the installation. In my case the partition is in sda7. It is said by the "dev/sda7"

Part 2

First of all launch the Partition Editor Software.

You can find the UUID of any of the partitions from right click the partition and select information at the Partition Editor Software.

Part 3

This part says where to mount the partition. Example is this sda7 partition has mounted as the home.

Part 4

This says about the file system about the partition. In this example case home partition contains the ext3 file format.

Part 5

I also don't know the exact meaning of this line. But it contains

 options       dump  pass

options can have values of errors=remount or defaults

errors=remount is used only for the root partition

But if u want u can use that option for any partition

Next two numbers represent dump and pass

It can have following variations

0 0 for /proc and swap

0 1 for root

0 2 for others

You can see that last two lines is bit different because they are for swap partition and cdrom. Don't edit them if unless you really needed and u know what to do there.

Most probably you will need to restart twice after editing this to make it work. Don't get afraid if it doesn't work at the first time.