Data Center Operations

What is Next-Gen AV?

2023-01-15T00:28:00.000+05:30

Next-generation antivirus (NGAV) refers to a new generation of endpoint protection software that uses advanced techniques such as machine learning, behavioral analysis, and cloud-based threat intelligence to detect and prevent malware, ransomware, and other types of cyber threats.

NGAV systems typically use a combination of signature-based and signature-less detection methods to identify and block malware. Signature-based detection uses known patterns or "signatures" to identify known malware, while signature-less detection uses techniques such as machine learning and behavioural analysis to identify unknown or "zero-day" malware.

NGAV systems also use cloud-based threat intelligence to stay up-to-date with the latest threats, and to share information with other systems to improve detection and response.

NGAV systems can also integrate with other security tools such as firewalls, intrusion detection systems, and endpoint detection and response (EDR) systems to provide a more comprehensive security solution.

NGAV has many advantages over traditional antivirus software, including a higher level of protection against unknown and advanced threats, faster response times, and better performance.

It's important to note that Next-gen AV is not a replacement for traditional antivirus software, but it can be used as a complement to it, to enhance the overall protection level for the organization.

Difference between Autosys and Crontab

2015-05-17T12:40:00.002+05:30

Autosys is a job scheduling tool that is used to manage and automate the execution of jobs, primarily in the context of an enterprise IT environment. It provides a centralized management console for scheduling, monitoring, and reporting on jobs and job streams. Autosys is more powerful and feature-rich than crontab and is more suited for enterprise-level scheduling needs.

Crontab, on the other hand, is a simple and lightweight scheduling tool that is built into the Unix/Linux operating system. It is used to schedule and automate the execution of commands or scripts on a regular schedule. Crontab is typically used to schedule system-level tasks such as backups, log rotations, and other maintenance tasks. Crontab is simpler to set up and use than Autosys, but it does not provide the same level of centralized management and reporting capabilities.

In summary, Autosys is a powerful and feature-rich enterprise-level scheduling tool, while Crontab is a simple and lightweight scheduling tool built into Unix/Linux.

MySQL basic commands

2015-04-26T00:07:00.002+05:30

Learn some basic commands for world's most popular MySQL database.

mysql -u [username] -p: This command is used to log in to a MySQL server using a specific username and password.

SHOW DATABASES;: This command displays a list of all the databases on the MySQL server.

USE [database_name];: This command is used to select a specific database to work with.

SHOW TABLES;: This command displays a list of all the tables in the currently selected database.

DESCRIBE [table_name];: This command is used to display the structure of a specific table, including the column names and data types.

SELECT * FROM [table_name];: This command is used to retrieve all the data from a specific table.

INSERT INTO [table_name] (column1, column2, column3) VALUES (value1, value2, value3);: This command is used to insert new data into a specific table.

UPDATE [table_name] SET column1 = new_value WHERE some_column = some_value;: This command is used to update existing data in a specific table.

DELETE FROM [table_name] WHERE some_column = some_value;: This command is used to delete data from a specific table.

EXIT;: This command is used to exit the MySQL prompt and return to the command line.

It's important to note that these commands should be used with caution, as some of them can have a significant impact on the data stored in the database.

The Best Scripting/Programming Language to learn

2014-04-04T11:05:00.002+05:30

The best scripting or programming language to learn depends on your goals and what you want to do with it. Here are a few popular choices and their uses:

Python: Python is a general-purpose, versatile and popular programming language. It's a great language for beginners as it has a simple and easy-to-read syntax. It is also widely used in scientific computing, data analysis, artificial intelligence, and back-end web development.

JavaScript: JavaScript is the language of the web. It's used to create interactive front-end web applications and is also increasingly being used on the back-end with technologies like Node.js.

Java: Java is a popular and versatile programming language. It is widely used to develop enterprise-level applications and is the primary language for Android mobile app development.

C#: C# is a language developed by Microsoft. It is primarily used to develop Windows desktop applications and games. It is also used to develop mobile apps and web applications using the ASP.NET framework.

Ultimately, it's best to learn the language that is most relevant to your goals and interests.

Autosys Troubleshooting

2014-03-03T23:55:00.002+05:30

There are a number of things that can go wrong with Autosys, a job scheduling tool, and the specific troubleshooting steps will depend on the issue at hand. Some common issues and troubleshooting steps include:

Job status not updating: Check the event server log for error messages and ensure that the event server is running.

Job not running: Check the job definition and make sure that the job is enabled and that the schedule and conditions are met.

Job running too often: Check the job definition and make sure that the schedule and conditions are correctly set.

Job running too long: Check the job log for error messages and performance issues.

Job not starting: Check the machine status and ensure that the machine is available.

Job not stopping: Check the job log for error messages and ensure that the stop command is being sent correctly.

Job not being reported: Check the alarm_if_fail report and ensure that the report is set up correctly.

Job not sending email notification: Check the mail server configuration and ensure that the mail server is running and that the job is configured to send email notifications.

It is also important to check the Autosys documentation and log files for more information on the specific error message and troubleshoot accordingly.

Openfiler storage installation in Vmware workstation

2014-02-25T23:35:00.001+05:30

Openfiler is a storage operating system made of standard Linux 2.6 kernel to support SAN and NAS appliance.It is also licensed under GPL license,hence it can be downloaded free of cost and use the same for testing as well as in production environments.Here in this post I have deployed Openfiler in vmware to learn more about storage technology and for testing.

Two versions are available in Openfiler

1.Open source version released with Standard features.
2.Commercial version released with Advanced features.(Availed with two types of subscriptions SME and Enterprise)

Openfiler also supports software and Hardware RAID with various file systems available in Linux market today Ext3,XFS,JFS and ReiserFS.
Storage spaces are exported via File level and Block level protocols.

File level protocols includes NFS,CIFS,HTTP/DAV and FTP.
Block level based protocols includes iSCSI and Fibre channel.

Storage can be accessed and configured through intuitive browser based configuration utility.

Advanced features available in commercial version are ,

1.Fibre channel target
2.High availability(HA)
3.Block level replication(includes WAN)
4.iSCSI target for Virtulization
5.Remote Disaster Recovery storage
6.Snapshot management

Openfiler is a Free storage Operating system available in market to test and deploy SAN and NAS appliances.In the below video we will see how to install openfiler in a vmware workstation.

Here I have used 32-bit version.You can also make use of the 64-bit latest version available in openfiler website to test the deployment and configuration.

Difference between rpm and yum in RHEL/Centos/Oracle Linux

2014-01-26T23:17:00.001+05:30

RPM (Red Hat Package Manager) and Yum (Yellowdog Updater Modified) are both package management tools used in Red Hat Enterprise Linux (RHEL), CentOS, and Oracle Linux to install, update, and remove software packages.

RPM is a low-level package management tool that is used to install, update, and remove individual software packages. It can be used to install packages from local or remote sources, and it works with files that have the ".rpm" extension.

Yum is a higher-level package management tool that is built on top of RPM. It is used to manage groups of software packages and their dependencies. It can automatically resolve dependencies and install all necessary packages, Yum can also automatically update all packages on the system, and it can also remove unnecessary packages. It works with repositories, which are collections of RPM packages that are stored on a server and can be accessed over a network.

RPM

1.Installation of packages through command line options.

# rpm -ivh apache 2.2.366.rpm

2.Dependency resolution is not possible

If we need to install apache ,all the dependency packages must be installed separately.

3.Group installation of packages is not possible

All the web server components cannot be installed as a group.

4. Centralized management of packages is not possible.

5. Cannot connect to online rpm repositories to install packages, each package must be downloaded.

6. RPM cannot check for the available software updates online and to install them automatically.

YUM

1. Installation of packages through keywords.

#yum install apache*

2.Dependency resolution is possible.

If we are installing apache ,all the dependency packages will be installed automatically.

3.Group installation of packages is possible

# yum groupinstall "Web Server"

4.Centralized management of packages is possible

All the packages can be stored on a centralized server and can be retrieved for installation of packages in one or more clients by updating the centralized yum repository in client yum configuration file.

5.Can connect to online rpm repositories to install packages,

6.YUM can check online for all the software updates online and install them automatically replacing the existing packages.

# yum install upgrade

In summary, RPM is a command line tool for managing individual packages, while Yum is a command line tool for managing groups of packages and their dependencies. Yum is easier to use and more efficient than RPM.

Additionally, Yum can be used to install and update packages from the official Red Hat and CentOS repositories, as well as from third-party repositories. This makes it an ideal tool for managing software packages in production environments, because it simplifies the process of keeping systems up to date with the latest security patches and bug fixes.

To configure rsyslog server and client in Linux

2014-01-26T22:27:00.003+05:30

In this post we will discuss how to configure rsyslog server and client in Linux. rsyslog utility is used to log the system activities in a remote system.This facility is used for troubleshooting any remote server issues ,that has been hacked by a hacker or in the situation like remote system goes down due to sudden hardware failures.

There are two methods commonly used.

1.Logging messages into the same server : All the system activities are stored in the local database(hard drive) of the server.

2.Logging message into the remote system : All the system activities are stored in the remote system ie on a centralized logging server, secured way to store critical messages.

To install rsyslog package in Linux(RedHat based systems)

# yum install rsyslog* [install in centralized server and client]
# chkconfig rsyslog on

Centralized rsyslog server(192.168.2.5) configuration

#vim /etc/rsyslog.conf

un comment line 13 and 14 (modload and udp) and save the config file.
[ Here the logs will be received from client through UDP protocol]

# service rsyslog restart

Client side configuration

# vim /etc/rsyslog.conf
*.* @192.168.2.5

save the configuration file.

# service rsyslog restart

To confirm the configuration from server end.

# cat /var/log/messages

Logs will be updated in client as well as in centralized server.

Disadvantage: All the logs file from clients will be in the same file.

To avoid create new directory for each client to redirect to the respective directory.
For more information on rsyslog configuration see /usr/share/doc/rsyslog-*/rsyslog_conf.html

Access Control List in Linux

2014-01-26T17:28:00.002+05:30

Files and directories have permissions by default,different permission to owner,group and others.These default permissions are set by the umask value.
To grant different permissions for different users, access control list is used. ACL is supported by RedHat Linux kernel with ext3,ext4 and NFS mounted file systems.ACL also recognized by samba file system shares configured on ext3 or ext4 file systems.

The copy and move commands also move the directory or file with the acl permissions.

To use acl, the partition must be mounted with acl support.Let see how to mount a ext4 partition with acl support

# mount -t ext3 -o acl device-name partition_name

To mount the partion permanently with acl support,make changes in /etc/fstab file.

# vim /etc/fstab

/dev/vol01/lv01 /work ext4 acl 0 0

By default NFS supports acl, if the exported file system is acl enabled.The NFS client can recognize the acl enabled share.

To disable acl on the NFS server, include no_acl option in the /etc/exports file.

To disable acl on NFS client side while mounting the NFS partition ,specify no_acl option in the command line or include no_acl option in /etc/fstab file .

To set acl in a file or directory for a user

# setfacl -m u:sam:rw /data

setfacl utility is used to add or modify acl for a file or directory.

m denotes modification
u usename(we can also give uid of the user)
rw permission granted for the user

In this example user sam is granted read and write access to /data directory.

To view the acl for a file or a directory

# getfacl /data

+ symbol at the end of the permission field, indicates file or a directory has acl in it.

To set different permissions for different users for a directory

# setfacl -m u:sam:rw,u:don:rwx /data1

To set acl for group in directory or a file

# setfacl -m g:hr:rw /data2

To set acl for others in directory or a file

# setfacl -m o:r /data2

To set acl recursively for all the files included in a directory.

# setfacl -R -m u:sam:rw /data

To apply default acl for a directory[acl will be applied to all the files and directories added in future.]

# setfacl -d -m u:susan:rx /data

To remove all acl for a directory

# setfacl -b /data
# setfacl -R -b /data (remove acl recursively)

Swap space management in Linux

2014-01-22T21:23:00.007+05:30

Swap space, also known as virtual memory, is an area on a hard drive that Linux uses to store data that would otherwise be stored in RAM. It is used when the system runs out of physical memory (RAM) and is used to temporarily store data that is not currently being used by the system.

Managing swap space in Linux can be done by using the following commands:

free : This command will show the current usage of swap space, including the total amount of swap space, the amount used, and the amount free.

swapon : This command will enable a swap partition or a swap file.

swapoff : This command will disable a swap partition or a swap file.

mkswap : This command will create a new swap file or partition.

swapon -s : This command will show the current status of swap space, including the location of the swap file or partition, the size, and the priority.

/etc/fstab : This is a configuration file that controls how swap partitions and swap files are automatically enabled at boot time.

It is important to monitor the swap usage, if the system is frequently using swap space, that means the system is running out of physical memory and it might be a good idea to add more RAM or to optimize the system's memory usage.

It's also important to note that if the system uses more swap space than physical memory it can cause a performance degradation.

Additionally, it's recommended to check the documentation of your specific Linux distribution for more information on swap space management and troubleshoot accordingly.

The recommended amount of swap space in Linux can vary depending on the specific use case and the amount of physical memory (RAM) available on the system. However, there are some general guidelines that can be used as a starting point:

For systems with less than 4GB of RAM, a swap space of 1-2 times the amount of RAM is recommended.

For systems with 4-16GB of RAM, a swap space of 2-4GB is recommended.

For systems with more than 16GB of RAM, a swap space of 8GB is recommended.

It's important to note that these are just general guidelines and the actual amount of swap space needed may vary depending on the system's usage and the specific applications running on it.

It's also important to note that with modern systems and Linux distributions, using swap space is not as frequent as before, thanks to the ability of the kernel to use other options like transparent huge pages and compaction as well as the use of more efficient algorithms for memory management.

In addition, if the system has a high I/O operations, it might be beneficial to have a swap partition on a fast storage medium.

It's recommended to monitor the system's swap usage and adjust the swap space accordingly and also check the specific documentation of your Linux distribution for more information on swap space management and troubleshoot accordingly.

Difference between SAN and NAS

2014-01-17T22:25:00.003+05:30

In this post I have listed main difference between SAN and NAS environment. To select the right technology for your infra structure ,find the advantages and disadvantages of each technology.

SAN (Storage Area Network) and NAS (Network Attached Storage) are both types of network storage, but they have different architecture, functionalities and are used for different purposes.

SAN (Storage Area Network) is a dedicated, high-speed network that provides block-level access to data. SANs use Fibre Channel or iSCSI protocols to connect servers to storage devices, typically disk arrays. SANs are typically used in enterprise environments to provide storage for applications such as databases, virtualization, and backups. SANs are typically faster than NAS systems, and are optimized for high-performance and high-availability environments.

NAS (Network Attached Storage) is a type of storage that connects to a network and provides file-level access to data. NAS devices typically use the NFS or SMB/CIFS protocols to connect to servers and client systems. NAS systems are typically used for file sharing and storage, and are often used in small-to-medium sized businesses and home networks. NAS systems are typically less expensive than SANs, and are optimized for ease of use and low cost.

In summary, SANs are designed for block-level storage, high-performance and high-availability and are used in enterprise environments, while NAS is designed for file-level storage, ease of use and low cost, and are used in small-to-medium sized businesses and home networks.

It's also important to note that in recent years the distinction between SAN and NAS is becoming blurry as some vendors are providing products that support both block and file level access, this is called SAN/NAS or unified storage.

NAS

Independent device
Uses standard protocols such as TCP/IP, CIFS,NFS, HTTP
Connected by Ethernet
NAS follows a client/server model
Best suited for File sharing applications

Advantages

High Storage capacity
Fast file access for multiple clients
Easy for data sharing

Disadvantages

Less efficient than SAN for moving large amount of data.

SAN

Network of storage devices
Uses FC , SCSI or iSCSI protocol for data transfer
Connected using Fiber Channel (FC) or SCSI over IP (iSCSI)
SAN provides direct access to the disks
Best suited for mission critical database applications

Advantages

Centralized Storage
Better disk utilization
Higher application performance

Disadvantages

High cost
Difficult to deploy and manage

Storage Basic Fundamentals

2014-01-17T21:48:00.001+05:30

Due to increased amount of data in enterprise business,storage has become most important to any business in the recent history.Cloud storage becomes popular due to its increased productivity and low cost in the market.In this post we can see some major storage concepts.

There are three types of storage

DAS - Directly attached storage
SAN - Storage Area Network
NAS - Network attached storage

DAS

In DAS, storage system(hard disk) is directly attached to server.This approach is seen in traditional desktop computers where hard disk have a dedicated connection to system motherboard.

Data is transferred through SCSI commands.Standard Buses used between the hard disk and computer are
SCSI, ATA and SATA

NAS

NAS is a file based storage, where storage resources are directly attached to LAN. NAS provides storage space to clients by sharing protocols such as NFS(Network File System) and CIFS(Common Internet File Sharing).

NFS - Commonly used sharing protocol in UNIX and Linux platform.
CIFS - Commonly used sharing protocol in Windows system.

NAS uses various network protocols such as HTTP,FTP and TCP/IP.

In NAS device, processor runs its own embedded operating system to provide access to shared storage.

SAN

SAN is a block based storage.It provides block based access to data through SCSI, FCAL/FCP and iSCSI protocols.
Block is a sequence of bits or bytes having nominal length.

Sharing of data is difficult in SAN.

In each server hardware HBA adapter is installed to provide block transfer of data through Fabric channel switches.

LUN - A disk provided by the SAN to client system that looks like locally attached storage.

Target - A system (SAN) that offers disk(LUN) to another client system to access storage.

Initiator - Client system that import the LUN provided by the SAN.Appropriate initiator software must be installed to access LUN.

Fabric - It is a collection of fiber channel switches ,that inter connects target and initiators.

Ports administration and Troubleshooting

2014-01-13T17:59:00.001+05:30

There are two types of ports

Software port: It is a piece of software in a machine used to communicate with remote application.

Hardware port: It is a physical peripheral in a machine used to connect systems and devices physically.

The combination of IP address and software port is known as socket number.

The port numbers vary from 0 to 65535(2 power 16=65535).Default ports or Well known ports range from 0 to 1023.

To view all the default port in Linux machine -> $ less /etc/services

Nmap(Network mapper), important monitoring tool used to scan open ports in local and remote machine.Nmap service can be installed in windows,Linux and solaris.

To install nmap command in Linux -> # yum install nmap

Command to scan for open ports in a machine(remote/local)

# nmap hostname

Command to check whether a particular port is open or not

# namp -p port_number hostname

Command to scan for entire network-ID/subnet mask

# namp IP-address/subnetmask

Command to check the protocol supported by the remote machine for communication

# nmap -so hostname

Command to scan for operating system details(remoet/local) with uptime

# nmap -o hostname

NFS-Server and Client configuration in Linux

2014-01-11T21:13:00.001+05:30

NFS- Network File System is the commonly used to share files over the network.It is widely used in UNIX-Linux based environments. NFS protocol is also used in Netapp storage for importing files to clients. NFS came across various improvements from verisons 1,2,3 and 4.
NFS 4.1 is in development stage.

NFS Server configuration in Linux[NFS version 3]

Command to install the NFS package

# yum install nfs*

Create export file to enter the NFS share details

# vim /etc/exports

/data/file1 *(ro,sync)

/home 192.168.1.0/24(rw,sync)

/data/folder1 *.test.com(rw,sync)

/data/database1 192.168.1.203/32(rw,sync)

Explanation:

Read-only access to the /data/file1 directory to all networks

Read/write access to the /home directory from all servers on the 192.168.1.0 /24 network, which is all addresses from 192.168.1.0 to 192.168.1.255

Read/write access to the /data/folder1 directory from servers in the test.com DNS domain

Read/write access to the /data/database1 directory from a single client 192.168.1.203.

Command to reload the NFS configuration file[show errors in config file, if errors available]

# exportfs -r

Command to restart the NFS services

# service nfs restart

# service nfslock restart

Command to start the NFS services on boot time

# chkconfig nfs on

# chkconfig nfslock on

To view the exported NFS mount points from the server

# showmount -e

NFS Client side configuration in Linux

To view the exported NFS shares from a particular server(192.168.2.1)

# showmount -e <192.168.2.1>

# mount -t nfs 192.168.2.1:/data/files /media

To mount the NFS share permanently after reboot

# vim /etc/fstab

Directory Mount Point Type Options Dump FSCK

192.168.2.1:/data/files /media nfs defaults 0 0

#mount -a

RPM commands in Linux

2014-01-11T20:38:00.000+05:30

RPM code is used for Linux like operating system ( RHEL , Centos ,Fedora) to build,query,install,update,verify and erase individual software packages.RPM package also contains detailed information about the package version modification,release date,depends on etc.Local rpm package database is stored at /var/lib/rpm ,contains information about file attributes and package dependencies.
Below I have listed some RPM commands used on day to day basis in Linux.

RPM file format: name-version-release.architecture.rpm

a) Command to install rpm package

# rpm –ivh package name

-i : install a package
-v : verbose output
-h: print hash marks as the package archive is unpacked.

b) Command to install package without dependency

# rpm -ivh package name - - nodeps

c) Command to install package with dependency

#rpm -ivh package name - - aid

d) Command to install a package forcefully

# rpm -ivh package name - - force

e) Command to upgrade a package

# rpm –Uvh package name

While upgrading ,original package(with exception of configuration files ) on the system will be removed and new packages will be installed. Configuration files from the original installation are saved with a “.rpmsave” extension.

f) Command to view all installed packages

# rpm –qa | less

g) Command to uninstall a rpm

# rpm –e package name

# rpm –ev - - nodeps package name

h) Command to check whether a particular rpm(vsftpd) is installed or not

#rpm –qa vsftpd*

i) Command to view document file for a particular rpm

#rpm –qd package name

j) Command to view configuration file for a particular rpm

# rpm –qc package name

k) Command to find the package name for a particular file name

# rpm –qf path to file

l) Command to view the information about the installed rpm package

# rpm –qi package name

m) Command to view the list of recently installed rpm

# rpm -qa --last | less

n) Command to view the dependency of rpm before installation

# rpm –qRp package name

o) Command to verify a installed package against rpm database

# rpm -Vp package name

To verify all packages installed

# rpm –Va

p) Command to import RPM GPG keys

# rpm - - import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 [Press tab to complete]

q) Command to view all RPM imported GPG keys

# rpm –qa gpg-pubkey*

r) Command to rebuild corrupted RPM database

#cd /var/lib/rpm
#rm __db*
#rpm –rebuilddb

# /usr/lib/rpm/rpmdb_verify Packages

To configure log on banner for ssh clients

2014-01-09T20:58:00.000+05:30

SSH(Secure shell protocol ) is used to connect UNIX based machines remotely. ssh protocol encrypts the Data connection between server and client. It uses port no 22.

While connecting servers remotely we can set banners to display in command line which includes server details and any access rights and issues. This is possible only in ssh protocol version 2 and initially no banners are configured.

Configuration details.

1.Create a banner file using vim editor and save the file.

$ vim /etc/banner

############ Welcome to Project X############
######You are accessing a secured environment###

2.Open the sshd configuration file (/etc/ssh/sshd_config) add the path to banner.

$ vim /etc/ssh/sshd_config

Banner /etc/banner

3. Restart the sshd service

$ service sshd restart

Now try to login to the server via Putty.You will see the banner configured.

Difference between NFS3 and NFS4

2014-01-09T20:44:00.000+05:30

Here I have listed some major differences between NFS3 and NFS4.Security has been major improvement taken into account for developing NFS4 version.Version NFS 4.1 is in development stage includes pNFS mechanism.

	NFSv3	NFSv4
Personality-Behavior	Stateless protocol	Stateful protocol
Semantics-support	Support UNIX only	Support UNIX and Windows
Authentication type	Weak -(AUTH_SYS)	Strong -(Kerberos)
Identification type	32-bit UID/GID	String based (xyz@blogspot.com)
Permissions type	UNIX based access	Windows like access
Transport protocol	UDP and TCP	TCP only
Caching data	Ad-hoc	Delegations

Autosys Job Scheduler workflow

2013-12-03T21:22:00.004+05:30

In the below post I have explained how Autosys job scheduler 4.5.1 works in a heterogeneous environment.

In most of the production environment event server and the event processor are installed on the same machine with required agents.

1.The event processor scans the event server database, every 5 sec to identify if any event ready to process.

2.If a STARTJOB event available in the event server and its starting conditions are satisfied( start time condition), the event processor extract the job attributes when and where the job should run from the event server(DB).

3.The event processor starts the remote agent on the client,where the job should be running.
In Unix machine inetd process starts the remote agent.
In Windows machine remote agent service is stared by the username and password available in the job definition of the job.

4.The event processor sends all job parameters to the remote agent and informs the event server about the status change of the job as STARTING.

5.Remote agent sends acknowledgement to the event processor that it has received all job attributes.Socket connection is terminated.At the same time event processor scans for other events in the event server(database).

6.Remote agent checks for the available resources in the server, execute the job and sends change status of the job as RUNNING to the event server.

7.Once the job is completed, remote agent sends SUCCESS/FAILURE status to the event server depends on the exit code of the job.Remote agent process is stopped finally.

8.If the job is SUCCESS ,the log file is deleted from tmp directory.If the job is in FAILURE status, error log file is saved to identify the issue.

Difference between ON HOLD and ON ICE in Autosys

2013-12-03T18:17:00.001+05:30

The difference between ON HOLD and ON ICE in Autosys is that,when a ON_HOLD job is taken OFF_HOLD,the job will be scheduled to run if its starting conditions are already satisfied.
But when a ON_ICE job is taken OFF_ICE ,the job will not run even though its starting conditions are met.The job is scheduled to run only when the starting conditions reoccur for this job.

When a job is ON_HOLD,its dependent jobs and downstream from this job will not run.
But when a job is ON_ICE ,the downstream from this job runs as though the job succeeded.

Monitoring Tools for Linux

2013-12-01T21:57:00.001+05:30

Here in this post I have explained few performance monitoring tools in Linux.These commands are very important for Linux system administrator to perform day to day activities to troubleshoot performance issues in server.

1. top

top, interactive command displays real time cpu,memory,process usage details.
Read more about top-->

2. sar

sar used to find the historical events in the server to troubleshoot cpu,memory,disk,network bottlenecks.
Read more about sar -->

3. free -mto

Used to view the physical memory and swap memory usage details.
m - displays results in MB
t - displays total(physical + swap) details.
o - used to hide other memory though-puts like buffers and cache.

Total physical memory - 339 MB

4.iostat

iostat displays information about disk I/O,CPU and remote disk information such as NFS. We can also view the statistics for a specific device.

5. vmstat

vmstat displays virtual memory statistics,memory usage including swap memory.
iostat and vmstat are part of sar utility in Linux.We have to install sysstat package to work with iostat,vmstat and sar utility.

6.w

Displays detailed information about the users logged in and the process used by the respective users.

Sar Command in Linux

2013-12-01T19:02:00.000+05:30

SAR (System Activity Monitor) command, frequently used monitoring command in Linux and Unix flavors. sar utility provided by the sysstat package in Linux.

In top we can view the real time analysis (CPU, Memory, load) when we logged into the server. But sar can be used to log the historical events in the server to analyze the bottlenecks with respect to CPU, Memory and load averages in the past.
sysstat package also installs other packages like iostst,pidstat,mpstat,sar etc..

sar utility also installs cron job function to record the historical events.

# vim /etc/cron.d/sysstat

The first cron job used to record the system events for every 10 minutes. Depends on requirement we can change it.
The second cron job runs at midnight to save the daily summary report.

We can also edit the sysstat configuration from the below file.

# vim /etc/sysconfig/sysstat

The HISTORY and COMPRESSAFTER parameters used to store the sar log files for a specified period of time.

To view the CPU usage of a system.

# sar 2 5

Displays 5 values of cpu usages with 2 sec interval.

To view the memory usage of a system

# sar -r 1 3

Displays 3 values of memory usage with 1 sec interval.

To view the server uptime

# uptime

The uptime command used to view
1. how long the server is running(uptime)
2.The no of users logged into the server
3.Load average of the server with interval 01,05,15 minutes.

Top command in Linux

2013-11-29T19:53:00.001+05:30

The top command in Linux used to display the real time process activity of a running system. By default it list the high CPU intensive task processes. Process values are changed every 3 sec by default.

Top command displays the system uptime, Memory/CPU/Swap usage, no of users logged in and total no of processes running.

Load average shows the running/runnable or uninterruptible processes on the Linux server. The three values show the load averages for the last 01, 05, 15 minutes.

Explanation:

PID – process id of the process
USER – process owner
PR – priority of the process
NI – nice value of the process
VIRT – total amount of virtual memory used by process
RES – total amount of non-swapped physical memory used by the process
SHR – total amount of shared memory used by the process.

Below are the options available in top command, give these options at the time of running the top command.

M – To arrange the processes by RAM utilization.
P – To arrange the processes by CPU utilization.
c – To display or hide command/process full path.
r – To renice a process. Press r and then give the PID (process id) to be reniced. Then give the renice value and press ENTER.
R – To arrange the processes by PID number.
h – To display other options available for top command.
1 – To display or hide the number of CPUs present on the Linux server.
k – To kill a process in server. Press k and then enter the PID of the process to be killed. Press ENTER finally.
q – To exit the top command.
s - To change the display time interval results of top command. Time interval is given in seconds.
W – To save the modified top command configuration.

Autosys Basic Commands

2013-11-25T18:49:00.000+05:30

Here I have detailed about Autosys basic commands to work with jobs .These commands helps us to perform day to day activities in Autosys. These commands can be given in Event server/Event processor through command line tools like Putty.

1.To print the status of the job(including last run schedule)
# autorep -J <job name>

2.To Print job details(job definition)
# autorep -J <job name> -q

3.To print the job last run details
# autorep -J <job name> -d

4.To print the job run history
# autorep –j <job name> -r -4 [only we can view last 5 status of jobs (0-4)]

5.To put job ON_HOLD
# sendevent -E JOB_ON_HOLD –j <job name>

6.To put job OFF_HOLD
# sendevent -E JOB_OFF_HOLD –j <job name>

7.To put the job ON_ICE
# sendevent -E JOB_ON_ICE –j <job name>

8.To put the job OFF_ICE
# sendevent -E JOB_OFF_ICE –j <job name>

9.To FORCE_START a job
# sendevent -E FORCE_STARTJOB -J <job name>

10.To kill a job
# sendevent -E KILLJOB -J <job name>

11.To mark job as success
# sendevent -E CHANGE_STATUS -s SUCCESS -j <job name>

14.To mark job as terminated
# sendevent -E CHANGE_STATUS -s TERMINATED –j <job name>

15.Command to find dependent job for job/box
# job_depends -c -J <box name/job name>

16.To print the Schedule details for a calender
# autocal_asc

Press Enter

Give the calander name

Press Enter

Enter P to print the details

17.To print job/BOX full details.[This will take more time to pull details]
# autorep -J <job name> -w

Autosys Tutorial

2013-11-25T16:36:00.001+05:30

In this Autosys tutorial I have explained about the Autosys architecture for version 4.5. Autosys job scheduler is an automated job control system for scheduling, reporting and monitoring. These jobs can reside on any configured machine (Windows, UNIX or Linux) that is attached to the network.
A job is any single command, executable, script file or Windows batch file.

Any status changes posted(updated) to the database are called Events.

Each job definition contains different job attributes and conditions, including when, where and at what time a job should run.

We can manage, monitor and pull report in Autosys using GUI (Graphical User Interface) and JIL (Job information language).JIL is a programming language defined for Autosys which has its own syntax.

Architecture of Autosys 4.5.1

Autosys job scheduler contains 3 major components. They are

1.Event server (Autosys database)

2.Event Processor

3.Remote agent

Autosys Scheduler has [Server-->Client] architecture and requires bundles SYBASE database or third party database software’s like Oracle or MYSQL database.

Event Server

Event server is the data repository for all events, job definition, reports and system information.

Event server is the database which stores all information including events and job definition.

Event Processor

Event processor is the heart of Autosys simply called as event _daemon simply a Unix process or windows service that is running in the server.

Event processor continuously scans the event server for any jobs or events to be processed. When event processor finds any events to be processed, it checks the starting conditions for the event. Depends on the attributes of the events or jobs, informs the appropriate remote agent to perform actions. The actions may be starting /stopping of jobs, checking of resources, monitoring of jobs etc.

We can setup a secondary event processor called shadow event processor. If the primary fails ,the shadow event processor starts processing the events without delay.

Remote agent

In a Unix machine remote agent is a temporary process started by the event processor to process the events in Autosys client machines.

In a windows machine remote agent is a temporary service started by the event processor to process the events in Autosys client machines.

Event processor starts the remote agent in a specified client as mentioned in the job/event attributes.

Once the job is received the by the remote agent, it process the task and sends the running, completion status to the event server and exit. If the remote agent unable to send the events to the event server, it repeat the communication with event server until success.