hippodata

NAS migration using Robocopy

2013-05-25T23:42:00.000-04:00

Robocopy - A command line utility to copy files from one location to other
Operating System - Windows (installed in 2008), older versions can download from here
Command - 'robocopy'

Highlights:
========
- Multithreaded copying
- Tolerates network interruptions and resume copying
- Skip NTFS junction points which can cause copying failures because of infinite loops
- Copy file data and attributes correctly, preserve original timestamps, as well as NTFS ACLs, owner information, and audit information using command line switches
- A "mirror" mode, which keeps trees in sync by optionally deleting files out of the destination that are no longer present in the source
- Skip files that already appear in the destination folder with identical size and timestamp

Limitations:
=========
- Robocopy will not copy open files
- Systems prior to Windows Vista do not mirror properly, they ignore changed security attributes of previously mirrored files

Syntax:
======
robocopy <Source> <Destination> [<File>[ ...]] [<Options>]

For more switches:

robocopy /?

Throttle Bandwidth - Switch is /IPG (Inter Packet Gap)

Formula - Delay (D, in milliseconds) required to simulate a desired bandwidth (BD, in kilobits per second), over a network link with an available bandwidth of BA kbps:

For example, on a 4Mbps line, you could throttle down a copy to use a maximum of 2Mbps by using the /IPG:125 switch.

Example:
=======
To copy all files (*.*) from Source 10.10.10.10 to destination 20.20.20.20

robocopy \\10.10.10.10\source\ \\20.20.20.20\destination\ *.* /mir

Note: If you mix up you source and destination all of your data will be deleted instead of copied.

To copy Only New Files and write to a log file

If you want to copy in stages and don't want to copy all the files over again to copy any new files that were added since the first copy you can use the "/XO" flag which excludes older files. This means if the destination file exists and is the same date or newer than the source file robocopy will skip that file.

robocopy x:\ y:\ *.* /e /w:0 /r:0 /secfix /xo /log+:c:\NAS_migration.log

/e - Copies subdirectories
/w - Specifies the wait time between retries, in seconds, default value is 30
/r - Specifies the number of retries on failed copies, default value is 1,000,000
/secfix - Fixes file security on all files, even skipped ones
/xo - Excludes older files
/log+ - Writes the status output to the log file

Scenario: Migrate NAS share by scheduling to Run Incremental Copies

Scheduling as below will keep the new NAS share up-to-date until the final cut-over is scheduled.

Step:1 Create a folder on the server

Run - cmd

mkdir c:\nas_migration

Step:2 Create a bat file with below command

robocopy x:\ y:\ *.* /e /w:0 /r:0 /secfix /xo /log+:c:\NAS_migration.log

Step:3 Add this to scheduled tasks

Step:4 Plan a cut-over window

During cut-over you may make the source 'read-only' and do the final incremental update to make sure the new share is mirror of source and let the system admins work to map the new share

Done!

Configure anonymous FTP access on VNX

2013-05-24T23:28:00.001-04:00

Step:1 Disable FTP

server_ftp server_2 -service -stop

Step:2 Log in to Control Station as root

Step:3 Change to /nas/sbin direcotry

cd /nas/sbin

Step:4 Add ftp user

/nas/sbin/server_user server_2 -add ftp

At prompt:
a. Any UID and GID, except -2
b. Home directory specified and available on Data Mover

Step:5 Disable ftp user's password

/nas/sbin/server_user server_2 -passwd -disable ftp

Step:6 Start FTP

server_ftp server_2 -service -start

Done!

Celerra to VNX CIFS migration

2013-05-24T23:03:00.001-04:00

Scenario: Migrate CIFS shares on Celerra to VNX

Tool: After experimenting with 4 different tools I recommend using Celerra Replicator as the best tool for this requirement

License: Get the free 180 day trial license for celerra replicator V2

Click here to learn about VDM

Step:1 Install temporary license

nas_license -create replicatorv2

Step:2 Setup communication

Source:

nas_cel -create remote_cel -ip 20.20.20.21 -passphrase nasadmin

IP: IP address of remote control station in slot 0

Destination:

nas_cel -create source_cel -ip 10.10.10.11 -passphrase nasadmin

IP: IP address of source control station in slot 0

Verify:

nas_cel -list

Step:3 Verify if all interfaces are able to communicate

server_ping server_2 -interface source_cel1 20.20.20.21

Repeat on all interfaces

Step:4 Setup interconnect between source server2 and destination server3

Source:

nas_cel -interconnect -create soure_cel1_s2s3 -source_server server_2
-destination_system remote_cel -destination_server server_3 -source_interfaces
ip=10.10.10.11,ip=10.10.10.12,ip=10.10.10.13 -destination_interfaces
ip=20.20.20.21,ip=20.20.20.22, ip=20.20.20.23 -bandwidth
MoTuWe08:00-13:00/8000

Note: If no bandwidth schedule is specified, all available bandwidth is used at all times.

Destination:

nas_cel -interconnect -create remote_cel1_s3s2 -source_server server_3
-destination_system source_cel -destination_server server_2 -source_interfaces
ip=20.20.20.21,ip=20.20.20.22, ip=20.20.20.23 -destination_interfaces
ip=10.10.10.11,ip=10.10.10.12,ip=10.10.10.13 -bandwidth MoTuWeTh08:00-17:00/3000

Validate:

nas_cel -interconnect -list
nas_cel -interconnect -validate id=30005

Step:5 Create a VDM replication session

nas_replicate -create rep_vdm1 -source -vdm vdm1 -destination -vdm dst_vdm -interconnect soure_cel1_s2s3 -max_time_out_of_sync 10

Note: If you specify a pool instead of an existing destination VDM, the pool is used to create the destination VDM automatically as read-only and uses the same name and size as the source VDM.

Verify:

nas_replicate -list

Step:6 Replicate the file system mounted to the VDM

nas_replicate -create vdm_v2 -source -fs src_fs_vdm -destination -pool id=3 -vdm
Vdm_2 -interconnect soure_cel1_s2s3

Note: Use the -vdm option used to indicate that the file system will be mounted on a VDM.

Verify:

nas_replicate -info vdm_v2

Look for last synchronization time

Step:7 Change the source file system to "read only"

Plan the cutover time to let users start using file system on new VNX; this is disruptive operation and should not perform until you are ready for brief outage associated with switchover period

1. quiesce or shutdown application that is using this file system
2. unmap the file system, change the properties to "read only" (better do it in unisphere)

Step:8 Create server interface on VNX

Failover from celerra to VNX by creating interface with same name as the source celerra server. If you are not sure of the name of source interface try executing the switchover from unisphere under "Replcias"

Source:

nas_replicate –switchover src_fs_vdm

After executing this command it will print a task id on the screen, please make a note of it.

Note: This command stops replication, mounts the source object as read-only, and marks the destination object as read-write sot that it can act as the new source object

Step:9 Check switch over status

nas_task -info 4211
nas_task -info all

Step:10 Verify file systems are available by testing from hosts

Step:11 Clean up replication sessions

nas_replicate –delete src_fs_vdm

Step:12 On VNX, back up file system data

Done!

Useful Celerra commands

2013-05-18T21:40:00.002-04:00

Command:1 Create file system

/nas/bin/nas_fs -name fs1 -type uxfs -create size=<initial size> pool=<from pool> -auto_extend yes -max_size <max size> -vp yes -option slice=y,mover=server_2

Command:2 Create Mount Point

/nas/bin/server_mountpoint server_2 -create /fs1

Command:3 Mount File System

/nas/bin/server_mount server_2 fs1 /fs1

Command:4 Enable Unicode for CIFS Compatibility

/nas/sbin/uc_config -on -mover server_2

Command:5 Start CIFS protocol

/nas/bin/server_setup server_2 -Protocol cifs -option start

Command:6 Create CIFS IP Interface

/nas/bin/server_ifconfig server_2 -create -Device <network device> -name CIFS -protocol IP <cifs interface ip> <netmask> <broadcast ip>

Command:7 Start CIFS IP Interface

/nas/bin/server_ifconfig server_2 CIFS up

Command:8 Add CIFS Server to serve UNC paths

/nas/bin/server_cifs server_2 -add compname=<hostname>,domain=<domain name>,netbios=<hostname>,interface=CIFS,local_users

Command:9 Export CIFS shares for usage

/nas/bin/server_export server_2 -name fs1 -option netbios=<hostname> /fs1

Command:10 Create Mount point for NFS

/nas/bin/server_mountpoint server_2 -c /fs1

Command:11 Export NFS Share

/nas/bin/server_export server_2 -Protocol nfs -option root=<ip network accessing export> /fs1

Command:12 Create replication interface -Source

/nas/bin/server_ifconfig server_2 -create -Device <network device> -name REPL -protocol IP <ip of replication interface> <netmask of replication interface> <broadcast address of replication interface>

Command:13 Start replication interface -Source

/nas/bin/server_ifconfig server_2 REPL up

Command:14 Create Link between control stations for replication communications -Source

/nas/bin/nas_cel -create <name of remote control station> -ip <ip of remote control station> -passphrase pass1

Command:15 Create replication interface -Destination

/nas/bin/server_ifconfig server_2 -create -Device <network device> -name REPL -protocol IP <ip of replication interface> <netmask of replication interface> <broadcast address of replication interface>

Command:16 Start replication interface -Destination

/nas/bin/server_ifconfig server_2 REPL up

Command:17 Create Link between control stations for replication communications -Destination

/nas/bin/nas_cel -create <name of remote control station> -ip <ip of remote control station> -passphrase pass1

Command:18 Create interconnect between primary and secondary control stations -Source

/nas/bin/nas_cel -interconnect -create IC_<localcsname>_<remotecsname> -source_server server_2 -destination_system <remotecsname> -destination_server server_2 -source_interfaces ip=<source replication interface ip> -destination_interfaces ip=<destination replication interface ip>

Command:19 Create interconnect between secondary and primary control stations -Destination

/nas/bin/nas_cel -interconnect -create IC_<localcsname>_<remotecsname> -source_server server_2 -destination_system <remotecsname> -destination_server server_2 -source_interfaces ip=<source replication interface ip> -destination_interfaces ip=<destination replication interface ip>

Command:20 Setup replication between control stations -Source

"/nas/bin/nas_replicate -create RP_<localcsname>_<remotecsname> -source -fs fs1 -destination -fs fs1 -interconnect IC_<localcsname>_<remotecsname> -source_interface ip=<source replication interface ip> -destination_interface ip=<destination replication interface ip> -max_time_out_of_sync 1 -background -overwrite_destination

Command:21 Show virtual interfaces (port channels) status

server_sysconfig server_2 -virtual -info trk0_fxg

Command:22 Set vlan tagging and mtu for interface

server_ifconfig server_2 <interface name> vlan=<number> mtu=<mtu>

Setup iSCSI storage on Celerra

2013-05-18T21:21:00.000-04:00

Step:1 Validate network interface on data mover

server_ifconfig server_2 -all

Step:2 Create and mount a file system called iSCSI

nas_fs -n iscsi -c size=100G pool=clar_r5_performance

server_mount server_2 iscsi /iscsi

Step:3 Create the celerra iSCSI target

server_iscsi server_2 -target -alias target -create 1000:np=172.12.32.189

Step:4 List iSCSI target

server_iscsi server_2 -target -list

Step:5 Create 2 iSCSI LUNs

server_iscsi server_2 -lun -number 1 -create target -size 1000 -fs iscsi

server_iscsi server_2 -lun -number 2 -create target -size 1000 -fs iscsi

#Size in MB

Step:6 List LUNs

server_iscsi server_2 -lun -list

Step:7 List all LUNs on target

server_iscsi server_2 -lun -info -all -target target

Step:8 Configure LUN mask

server_iscsi server_2 -mask -set target -initiator <fqdn> -grant 1-2

#Replace <fqdn> with the iscsi initiator

Step:9 Start iSCSI service

server_iscsi server_2 -service -start

Done!

Virtual Data Mover (VDM)

2013-05-18T20:53:00.000-04:00

What is a VDM?
=============
-It is a feature that allows the separation of CIFS servers and their associated environment into virtual containers
-Enables replication of CIFS environments from a primary site to a secondary site without impacting server access
-Allows the movement of CIFS servers from one Data Mover to another
-VDMs store dynamic configuration data such as; local groups, shares, security credentials, audit logs, home dir information, kerberos

Benefits:
========
-Easier management of CIFS servers
-Provides server isolation and security
-CIFS servers within a VDM can be moved from one physical data mover to another without changing the share configurations

Key Concepts:
=============
-A VDM can contain one or more CIFS servers
-Atleast one interface must be available for each CIFS server that is created
-CIFS servers in each VDM have access only to the filesystems mounted to that VDM
-In contrast, a physical data mover supports NFS and CIFS servers
-The UserMapper configuration is at the physical data mover level
-VDMs exist within a directory structure on a physical data mover's root file system, with VDM configuration information being stored in the root file system

Four states of VDMs:
===================
Loaded: Fully functional active VDM with file system being loaded in read/write state
Mounted: In this state, the file system is mounted read only and CIFS servers are not active
PermUnloaded: The VDM file system is not mounted on the Data Mover
TempUnloaded: This is used to stop activity temporarily in preparation for replication

Creating a Loaded VDM:
=====================
Step:1 Create a VDM in a loaded state with name 'vdm02'

nas_server -name vdm02 -type vdm -create server_2 -setstate loaded

To create a VDM in a loaded state with root file system of default size being 128 MB

nas_server -name vdm02 -type vdm -create server_2 -setstate loaded pool=clar_r5_performance

Step:2 Mount the above created file system

Create mountpoint for the file system in VDM

server_mountpoint vdm02 -create /mntvdm

Mount the file system

server_mount vdm02 fs01 /mntvdm

Step:3 Configure CIFS for VDM

server_cifs vdm02 -add compname=hippodata,domain=corp.hippodata.com,interface=cge0-2

Step:4 Join the CIFS server to the domain

server_cifs vdm02 -join compname=hippodata,domain=corp.hippodata.com,admin=administrator

Step:5 Create CIFS shares

server_export vdm02 -protocol cifs -name vdmroot /mntvdm

Moving a VDM:
============
To move a VDM from one data mover to another

nas_server -vdm vdm02 -move server_3

For this to happen:
-target data mover should have identical interface names to the source
-target data mover should not have CIFS servers with the same name as the source

Remane a VDM:
=============
nas_server -rename vdm02 vdm01

Deleting a VDM:
=============
nas_server -delete vdm01

NetApp Cluster-mode networking deep dive!

2013-05-18T15:29:00.000-04:00

While administering NetApp cluster-mode, it's very important to understand networking and related concepts for successful troubleshooting in case of network failures. In this post I will be discussing the key network concepts related to cluster-mode.

Networking has gained special focus in cluster-mode with a goal to integrate fully qualified, high-performance switches into the sacle-out cluster and management networl

Key concepts:
==========
-Each controller have multiple physical network ports depending upon the controller model
-Physical network ports and logical interfaces (LIFs) are two different entities
-SAN environments use HBA ports as data ports

Roles:

Cluster ports:
==========
-Defaults to two ports for each node
-These ports are reserved strictly for the intracluster traffic

Data ports:
========
-Defaults to two ports for each node
-These ports are used for client data access and for the cluster management LIF
-These can NICs or FC ports depending upon the NAS or FC SAN environments

Node-Management ports:
===================
-Defaults to one port for each node
-These ports are used by administrators to configure the node
-Can be connected through SSH or a web-browser

Intercluster ports:
=============
-Defaults to none for each node
-These are used to communicate with another cluster
-These can be failed over to data ports
-Data ONTAP cluster-mode has the ability to do intercluster (or cross-cluster) data-protection mirrors that is performed over ports that have either the role of intercluster or data

To view available network ports:

net port show

To view available fiber-channel ports:

network fcp adapter show

Interface group (ifgrp):
=================
-Same as a virtual interface (vif) and port trunk
-It combines two or more physical ports and presents them as single logical port
-ifgrp provides redundancy and more bandwidth

Logical Interface (LIF):
=================
-IP addresses are associated with network ports; many data IP addresses can be assigned to a single network port
-A node can have a maximum of 128 LIFs
-Contains one node management LIF per node that cannot be failed over or migrated
-Contains one cluster management LIF per cluster that can be failed over or migrated throughout the cluster
-Contains two cluster LIFs per node that can be failed over or migrated within its node
-Contains multiple data LIFs per data port to face the client allowing NFS, CIFS, iSCSI and FCoE that can be migrated or failed over throughout the cluster
-FC LIFs use WWPNs

To view available logical interface ports:

network interface show

LIF roles compatibility with port roles:
=============================
Data----------------->Data
Cluster-------------->Cluster
Intercluster--------->Intercluster, Data
Node-management------>Node-management, Data
Cluster-management--->Node-management, Data

LIFs routing:
=========
-Configure routing groups and static routes to control LIFs in a Vserver to use network for outbound traffic
-A routing table contains each LIF association with one routing group to use the routes of that group
-Multiple LIFs can share a routing group
-A static route can be defined between a LIF and a specific destination IP address

To view the available routing-groups:

network routing-groups show

To view the route information:

network routing-groups route show

LIF failover - Automatic failover during outage or reboot
LIF migrate - For manual failover scenario
LIF revert - Manually or automatically sending LIF back to its home

Note: LIF failover is controlled by the failover policies and failover rules

LIF failover policies:
==============
nextavail: The LIF moves to the next available working port in a round-robin fashion
priority: The LIF moves to the port or node as specified
disabled: LIF does not failover

To view the LIF failover policy:

network interface show

->look for failover policy

To view the summary of failover ports:

network interface failover show

Optimized and Nonoptiomized paths:
===========================
-Optimized paths are the primary or favored paths that are active between the initiator and target that have same latency level
-NonOptimized paths are the secondary or unfavored paths that are inactive between the initiator and target that have different latency level

For us to provide multipath access there are two available choices:

Symmetric: In this all paths are favored or optimized
Asymmetric: In this only certain paths are favored or optimized

Choosing optimized paths can be done manually (you don't want to do it!) or automatically by using ALUA (asymmetric logical unit access)

ALUA:
======
-Also known as Target Port Group Support (TPGS) in the SCSI primary commands (SPC) standard
-ALUA can operate in the following possible states:

a. Active/optimized
b. Active/nonoptimized
c. Standby
d. Unavailable

Replicated database - NetApp cluster-mode

2013-05-18T14:25:00.002-04:00

The RDB is the 'vol0' volume which is created on every node of the cluster to manage the cluster operations. No user data is contained in it and is not available for client visibility. RDB's are made up of four units namely VLDB, Vifmgr, management and BCOM.

Key features of RDB:
=================
-Cluster with 'n' nodes will have 'n' RDB's where in one of it will be the master and the rest are secondary
-RDBs are synchronized and the operations are transactional; which means the entire transactions are either committed or rolled back
-RDB database reads are performed locally on each node, but an RDB write is only performed to "master" RDB database and the changes are replicated to other databases in the cluster
-The master is elected by the members of the cluster management units; a secondary can become a master and viceversa when master has communication issues
-One node in the cluster has a special tie-breaking ability called 'epsilon', unlike the master that may be different for each RDB unit, epsilon is a single node that applies to all units

VLDB:
=====
-This is one of the RDB unit that runs as 'vldb'
-Contains the index information of which the D-blade currently owns a volume and currently serves an aggregate
-The content of VLDB is cached in memory on each node for instant access by each N-blade and SCSI-blade

Management:
==========
-It is the M-host, which enables management of the cluster from any node
-Runs as the management gateway daemon - 'mgwd' on every node
-The information contained is used by the CLI and element manager

VifMgr:
=====
-Runs as 'vifmgr'
-Contains LIF configuration and failover policies

BCOM (blocks configuration and operations manager):
=========================================
-Runs as 'bcomd', to manage the configuration of SAN data access
-Contains LUN maps and initiator groups

Quorum:
=======
-It is a simple majority of connected, healthy nodes
-There will be a cluster-wide quorum and an individual unit being in or out of quorum
-When a unit goes out of quorum, reads from the unit can still occur, but not the writes
-The voting determines which node will be their master; so if a node is taken down for an extended period of time, it should be marked as ineligible so the quorum is not affected

system node modify -node <node> -eligibility false

Dataflow in NetApp cluster-mode

2013-05-18T13:20:00.001-04:00

To understand the dataflow in cluster-mode, one needs to understand the four major software components along with other key software components on every node in the cluster, they are as follows:

a. N-blade, network "blade"
b. D-blade, data "blade"
c. SCSI-blase, SCSI SAN "blade"
d. M-host, Management blade

Other components:
==============
-The cluster session manager (CSM)
-RDB units, which is the volume location database

All the above mentioned components are accessed only by well-defined application APIs. The N-blade and SCSI-blade translates client requests into Spin Network Protocol (SpinNP) requests. THe D-blade contains the WAFL file system that handles the SpinNP requests. The CSM is the SpinNP layer between the N-blade, the SCSI-blase and the D-blade.

Key Concepts:
============
N-blade: The N-blade handles the networking, NFS and CIFS requests and then translates to SpinNP requests as inputs to CSM

SCSI-blade: The SCSI-blade handles the networking, FC, FCoE and iSCSI requests and then translates to SpinNP requests as inputs to CSM

CSM: CSM then acts as a communicator between N-bade or SCSI-blade SpinNP requests and input them to the D-blade over UDP/IP

D-blade: The D-blade which contains the WAFL file systems then handles the SpinNP requests and them communicate disks and tape devices using FC

Dataflow on a local node:
===================
1. NAS or SAN clients sends the write requests to a data logical interface (LIF)
2. N-blade (NAS) or SCSI-blade (SAN) which are associated with that LIF translates the NFS or CIFS, FC, FCoE, or iSCSI request to a SpinNP request
3. The SpinNP requests then passed to the D-blade via CSM
4. D-blade then sends the data to NVRAM and to the disks
5. The acknowledgments then works its way back to the client

Dataflow on a Remote node:
=====================
1. NAS or SAN clients sends the write requests to a data logical interface (LIF)
2. N-blade (NAS) or SCSI-blade (SAN) which are associated with that LIF translates the NFS or CIFS, FC, FCoE, or iSCSI request to a SpinNP request
3. The SpinNP requests then passed to the "remote" D-blade via CSM
4. D-blade then sends the data to NVRAM and to the disks
5. The acknowledgments then works its way back to the client

Vserver and Flex Vol creation - NetApp Cluster-mode

2013-05-18T02:21:00.000-04:00

Scenario:
======
a. Create a cluster virtual server (Vserver)
b. Create a flex vol

Step:1 View the available Vservers in the cluster

vserver show

"node" Vservers are the scope where vol0 volumes exist and the node management LIFs and cluster LIFs of each node. The "admin" Vserveris the scope in which the cluster management LIF exists

Step:2 Create a Vserver with name vs2

vserver create -vserver vs2 -rootvolume vs2root -aggr node1_aggr1 -ns-switch file -rootvolume-security-style unix

With the above command we are creating a Vserver vs2 along with a root volume 'vs2root'. vs2root is th root volume (root of this namespace) of this Vserver.

Note: Vservers are independent of physical nodes and so even though'vs2' resides on 'node1' it does not essential mean it is assigned to node 'node1'

Step:3 Verify

vserver show

Step:4 View summary of the Vserver just created

vserver show -vserver vs2

Step:5 View list of volumes

volume show

Note: The Vserver root volume that was just created appears

Step:6 View default volume size

With the 'vserver create' command we cannot specify the volume size, there by this value is defaulted; however, the size can be changed with 'volume modify' command.

Step:7 View all the attributes of the volume just created

volume show -vserver vs2 -volume vs2root

Note: The slash (/) signifies that this is the root volume of this namespace.

Step:8 (b) Create a flex vol

volume create -vserver vs2 -volume volume2 -aggr node1_aggr1 -junction-path /vol2

Step:9 Verify

vol show

Step:10 View summary of the new volume

vol show -vserver vs2 -volume volume2

Done!

NetApp Cluster-mode aggregate creation

2013-05-18T02:00:00.000-04:00

Scenario:
======
a. Create a new aggregate with name node1_aggr1 on node1
b. Add disks to the aggregate

Step:1 View the available aggregates

storage aggr show

Step:2 View the avaialable disks

storage disk show

Notice the disks that belongs to aggregates and the spare disks that are available to create additional aggregates

Step:3 Create an aggregate

storage aggr create -aggr node1_aggr1 -node node1 -diskcount 6

Step:4 Verify

stor aggr show

Step:5 Detail view of the new aggregate

stor aggr show -aggr node1_aggr1

Step:6 (b) Add disks to the aggregate

aggr add-disks -aggr node1_aggr1 -diskcount 10

Step:7 Verify the number of disks

aggr show -aggr node1_aggr1 -instance

Done!

Run SPcollects on Clariion - Basics

2013-05-18T01:35:00.000-04:00

SPcollects contain detailed log and configuration data for a CLARiiON Storage Processor. These are used by EMC Support to diagnose faults and configuration problems. Gathering these files when creating a Service Request is highly recommended to troubleshoot a problem. SPcollects should also be retrieved when gathering data for a performance analysis, along with NAR files.

An SPcollect has the filename format of:

Serial Number _ SP A/B _ Year_Month_Day _ Time _ signature _data.zip

A file that ends with runlog.txt, indicates that an SPcollect is still running and this will disappear once the SPcollect file completes.

Navisphere CLI:
=============
Step:1 Open a command prompt on the Management Station.

Windows: cd c:\program files\emc\navisphere cli

Unix: export PATH=$PATH:/opt/Navisphere/bin

Step:2 Run SP collect

naviseccli -h <SP_IP_address> spcollect

This starts the SPcollect script.

Wait at least 10 minutes for the SPcollects to run, before attempting to retrieve them.

Step: 3 Verify

naviseccli -h <SP_IP_address> managefiles -list

This will list the files created by SPcollect.

Step:4 Copy the files to management station

naviseccli -h <SP_IP_address> managefiles -retrieve

This will display the files that can be moved from the SP to the Management Station.

Enter files to be retrieved with index separated by comma OR by a range OR enter 'all' to retrieve all file OR 'quit' to quit

Step:5 Upload to EMC support

Windows: Very intuitive

Unix: Use Winscp to copy from management station to local work station and upload to EMC suppport

Enhancements to NFS and SMB - NetApp Cluster-mode support

2013-05-17T11:03:00.001-04:00

NFSv4:
=====
1. Data ONTAP 8.1 cluster-mode introduces support for NFSv4 protocol specification as well as elements of NFSv 4.1
2. cluster mode continues to fully support NFSv2 and NFSv3 although you should not use NFSv2 with cluster mode
3. NFSv4 support brings the Data ONTAP 8.1 cluster mode operating system in parity with the Data ONTAP 7.3 operating system
4. The key feature of NFSv4 is referrals, NFSv4.1 is a minor revision of version 4.0 and is an extension of version 4 not a modification, so it's fully compliant with the NFSv4 specification it extends delegations beyond files to directories and send links introduces NFS sessions for enhanced efficiency and reliability provides parallel NFS, pNFS

Remote file access:
==============
-It is defined as the file access in which a client connected to a logical interface LIF going to a physical port on one controller accesses a file that is hosted on a different controller in the same cluster
-Remote file access has traditionally been a performance concern for clients which has been fixed with Data ONTAP Cluster mode operating system

Scenario:
=======
A client is mounted to a Data LIF that is hosted on node1 and has a file operation with the destination in a volume on node4 the request is serviced by the node1 protocol stack, that protocol stack looks for the location of the volume and directs the operation to node4; which hosts the volume. The request traverses the cluster network and the result is returned to the client along the same path

With pNFS when a file is open by an NFS client that mounted data LIF on node1 serves as the metadata path, because this is the path that will be used to carry out discovery of the target volumes location. If the data is hosted by node1 the operation will be managed locally in this case the local node discovers that the data is on node4, based on the pNFS protocol the client will be redirected to a LIF hosted on node4 the request as well as subsequent requests to the volume are serviced locally bypassing the cluster network

when a volume is moved to an aggregate on a different node the pNFS client data path is redirected to a Data LIF hosted on the destination node

To enable pNFS:

cluster1::> vserver nfs modify -v 4.1 -pnfs enabled

Note: Clients should be able to support pNFS, it is supported with RHEL 6.2 and Fedora 14.

NFSv4 referrals and pNFS do not work together. By keeping network accessing data local pNFS reduces the amount of traffic that traverses is the cluster network. Unlike with NFS referrals, pNFS works seamlessly to the client it does not require a file system remount to ensure an optimized path with pNFS because the network redirect does not happen at Mount time. The final handle will not be left stale when a volume is moved to an aggregate on a different node

SMB 2.0 and SMB 2.1
=================
1. In addition to the SMB 1.0 protocol, the Data ONTAP cluster-mode operating system now supports SMB 2.0 and SMB 2.1.
2. SMB 2.0 was a major revision of the SMB 1.0 protocol including a complete reworking of the packet format SMB 2.0 also introduces several performance improvements relative to previous versions.
3. Efficient network utilization request compounding which stacks, multiple SMB into a single network packet larger read and write sizes to exploit faster networks File and directory.
4. Property caching durable file handles to allow SMB connection to transparently reconnect to the server if a temporary disconnection occurs such as over a wireless connection.
5. Improve message signing with improved configuration and interoperability with HMAC SHA-256 replacing MD5 at a hashing algorithm SMB 2.1.
6. Provides important performance enhancements, these enhancements include the following -client opportunistic lock (Opplock leasing model)
-large maximum transmission unit MTU support
-improved energy efficiency for client computers support for previous versions of SMB
7. The SMB 2.1 protocol provides several minor enhancements to the SMB 2.0 specification 8. Data ONTAP 8.1 cluster mode supports most but not all of the SMB 2.1 features.
9. The following SMB 2.1 features are not supported
-large MTU resilient handles
-branch cache support for SMB 2.1 is automatically enabled when you enable the SMB 2.0 protocol on a virtual server (Vserver)

Use the following command to enable SMB to point out for the server:

cluster1::> vserver cifs options modify -vserver vs1 -smb2 -enabled true

Features of SMB 2.1 Leases:
=====================
-File and metadata caching
-Reduces bandwidth consumption
-Retention of cached data after a file is closed
-Full caching with multiple handles as long as those handles are opened on same client

Basics of NetApp Cluster-mode Shells

2013-05-17T01:44:00.001-04:00

Similar to 7-mode we have three administrative privilage levels: admin, advanced and diagnostic. Clustered ONTAP system can be managed either from CLI or GUI (system manager or element manager). There are three different shells available with different scopes: Clustershell, nodeshell, systemshell

Step:1 Access through CLI

Clustershell:
=========
-SSH is default method
-One can access clustershell using cluster management LIF (recommended) or the node management LIFs
-Scope: entire cluster

LIFs Management: 'net int show' with display the output of available cluster management and node management LIFs along with their IP's, current node, current port and status

Nodeshell:
=======
-Access nodeshell within the clustershell
-Scope: limited to one node at a time
-Command to access nodeshell is:

system node run

Systemshell:
=========
-Access systemshell from within the clustershell
-This is to access BSD shell
-Scope: single node at a time
-To access systemshell:
a. security login unlock -username diag
b. system node systemshell
c. Login as "diag"

Step:2 Access through GUI

-Start the system manager client using the IP address of the cluster management LIF
-This is well known interface to all the existing 7-mode administrators and I don't think more explanation is needed here

Steps to setup NetApp Cluster-mode

2013-05-17T01:03:00.003-04:00

Step: 1 Hardware setup

a. Connect controllers to disk shelves (FC connectivity)
b. NVRAM interconnect to high availability cable between partners (10GbE or infiniBand)
c. Connect controllers to network such that each node have exactly two connections to the dedicated cluster network, at least one data connection. Also the well known RLM connection for troubleshooting purposes when needed.

Note: Cluster connections must be on a network with dedicated cluster traffic, where as data and management connections are on a distinct network.

Step:2 Power-up

a. Power up network switches
b. Power up disk shelves
c. Power up storage controllers

Step:3 Firmware

a. During boot process press any key to enter the firmware
b. Two compact flash images: flash0a and flash0b are available. To 'flash' (put) a new image on primary flash one needs to configure management interface.

Note: For auto option of ifconfig, DHCP or BOOTP server must be available on management network. If it doesn't one must run ifconfig <interface> addr=<ip> mask=<netmask> gw=<gateway>

c. Once the network is configured, ping to test and flash the image; run flash tftp://<tftp_server>/<path_to_image>flash0a

Step:4 Installing ONTAP 8.1

a. Run option 7 to install new software first
b. Enter a URL to ONTAP 8.1 tgz image
c. Allow the system to boot when complete

Note: One can type boot_primary if node stops at firmware prompt

Step:5 Initialize a Node

a. Run option 4
b. This initialization clears the three disks that the system uses for the first aggr that it creates and a vol0 root volume on it
c. This must be run on both nodes of each HA pair

Step:6 Cluster setup wizard

a. From boot menu, boot normally and login as "admin" with no password
b. The first node creates the cluster
c. The following information is required for the setup:
-Cluster name
-Cluster network ports and MTU size
-Cluster base license key
-Cluster management port, IP address, netmask, and default gateway
-Node management port, IP adress, netmask, and default gateway
-DNS domain name
-IP address of DNS server
d. Subsequent nodes join the cluster

Step:7 Normal boot sequence

a. Firmware loads the kernel from CF
b. Kernel mounts "/" root image from rootfs.img on CF
c. Init is loaded and startup scripts run
d. NVRAM kernel modules gets loaded
e. Tmgwd is started
f. D-blade, N-balde and other components are loaded
g. vol0 root volume is mounted from local D-blade
h. CLI and element manager are ready for use

Step:8 Create a cluster

cluster create -license <name> -clustername <clustername> -mgmt-port <port> -mgmt-ip <mgmtip> -mgmt-netmask <mgmtnetmask> -mgmt-gateway <magmtgateway> -ipaddr1 <clusterip1> -ipaddr2 <clusterip2> -netmask <clusternetmask> -mtu 9000

Step:9 Join a cluster

Run this command from the node that wants to join the cluster

cluster join -clusteripaddr <remoteclusterip> -ipaddr <clusterip1> -ipaddr2 <clusterip2> -netmask <clusternetmask> -mtu 9000

Step:10 Licenses

a. Base
b. NFS
c. CIFS
d. iSCSI
e. FCP
f. SnapMirror_DP
g. SnapRestore
h. Flexclone

Note: One can add licenses in the cluster shell; system license add

Step:11 NTP

a. NTP is disabled by default and needs manual set up of date, time and time zone

system date modify

b. Verify and monitor

system services ntp config show
system services ntp server show

Basics of NetApp Data ONTAP Cluster-mode

2013-05-16T23:33:00.001-04:00

With ever increasing business needs the amount of data need to be stored, managed and backed up is getting complicated. To handle this kind of requirements industry has two models: scale up and scale out.

Scale up:
======
1. limits performance and capacity
2. additional controllers are managed independently and do not provide any shared resources
3. not the best model for larger environments

Scale out or clustering:
=================
1. no limit on performance and capacity
2. additional controllers are added seamlessly to take advantage of the shared resources
3. all the operations including hardware replacements, upgrades and almost everything can be done non-disruptively
4. NetApp Data ONTAP cluster mode supports both SAN and NAS environments along with features like thin provisioning, deduplication, compression and data replication
5. Unified architecture with ability to support multiple data access protocols like FC, FCoE, iSCSI, NFS, pNFS, CIFS

Non Disruptive Operations:
=====================
1. Businesses today have rigid requirements and downtime cannot be tolerated.
2. Data ONTAP Cluster-Mode was built in with NDO as its fundamental feature to keep storage infrastructure remain up and to serve data during hardware and software maintenance and refresh operations
3. Data volumes can be move non-disruptively with data motion across the cluster at any time

NetApp 7G to Cluster-mode transistion:
==============================
It has always been a big gap for netapp to merge both the products, but with ONTAP 8 it allowed 7G customers to run ONTAP 8 in 7-mode while giving first step in the eventual move to a clustered environment. ONTAP 8.1 OS is a major step in merging 7G and cluster-mode.

Difference between 7G and Cluster-mode:
===============================

In ONTAP cluster-mode systems, each node has a vol0 root volume similar to the 7G systems. In addition, each Vserver has a root volume that is the root of the namespace that can be composed of many volumes spread throughout the cluster

7-mode features not in Cluster-mode:
============================
1. MetroCluster
2. SnapVault
3. Snaplock
4. Synchronous snapmirror, Qtree Snapmirror
5. Volume snapmirror between 32-bit and 64-bit aggr
6. Virtual storage console, SMVI
7. DataMotion for vFiler
8. SyncMirror
9. NetApp storage encryption disks
10. IPv6

Cluster-mode features not in 7-mode:
===========================
1. LIF migration and failover
2. Cluster management as a single system
3. SMB 2.1
4. NFSv4.1, pNFS
5. Asynchronous volume-level replication
6. Load-sharing SnapMirror functionality
7. Onboard antivirus scanning (McAfee and Sophos)

Scalability:
========
1. 2 to 24 nodes, by looking like a single file system
2. Throughput scales linearly to multiple gigabytes per second
3. Can be scaled to petabytes
4. Online load balancing and scaling
5. High-performance computing (HPC)

Flexibility:
========
1. Mix of FC, SATA and SAS drives
2. NDU data movement between tiers

Note: data doesn't need to be moved between tiers for clients to be able to access it. FAS 6200 series can access the SATA storage just as easily as they access the FC and SAS storage. The reason to use tiered storage within a cluster is to improve performance, not accessibility.

Cluster Resources:
==============
1. Three types of Vservers: nodes, cluster, administration
2. Nodes
3. Aggregates
4. Volumes
5. Namespaces

Node Vserver:
===========
-Physical node
-Associated with cluster LIFs and node management LIFs

Administration Vserver:
==================
-Physical cluster
-Associated with cluster management LIF

Cluster Vservers:
=============
-Not associated with any single node
-Contains resources within scope: Namespace, volumes, data LIFs, protocol servers (NFS,CIFS,FC,FCoE and iSCSI)

Note: When a cluster is created, the administration Vserver is automatically created, where as a node Vserver is automatically created when a node is joined to cluster. Cluster Vservers are created by administrators to build global namespaces.

How a qtree be accessed by both Unix and Windows - NetApp

2013-05-15T23:40:00.002-04:00

Caution: I don't recommend using mixed mode for different reasons. Use UNIX style!

Step:1 Create a Unix qtree on the filer and export it for Unix clients
filer> qtree create /vol/vol0/share
filer> exportfs -o root=<unix_server> /vol/vol0/share

Step:2 From the Unix server mount the filer's share
unix> mount filer:/vol/vol0/share /share

Step:3 On the filer share the resource for Windows users
filer> cifs shares -add share /vol/vol3/share

Step:4 From the Windows computer map the filer's resource
Right-click on My Computer > Map the network drive > enter the filer's name\share

Step:5 On the filer edit /etc/usermap.cfg and add the following entries for each user:
win_domain\userA => unix_userA

When a Windows userA tries to access the file on the filer's share, the filer as per the entry in /etc/usermap.cfg forwards the request to the unix, and the unix userA permissions are assigned to the windows userA. So all you need to do is, assign the proper permission to unix userA which in turn is applied to the respective Windows user.

Example:
=======
filer> wrfile /etc/usermap.cfg
"hippo data" == hippo

The Windows name hippo data maps to the UNIX name hippo and vice versa.

UNIX host to access data with NTFS security style using NFS

2013-05-15T23:24:00.000-04:00

Step:1 User login from the UNIX host, the host requests user and group information for the user from the name services configured in the /etc/nsswitch.conf file. The data can be retrieved from local files, an NIS server, or an LDAP server.

Step:2 The configured name service returns user and group information to the UNIX host.
All user information needed to log in, including UID, GID, and the user shell and home directory, is returned. Additionally, the user’s secondary group GIDs are returned.

Step:3 Using the user information retrieved from the identity store, the user is authenticated and allowed access to the UNIX host. The user and group information retrieved in step 1 is cached and used to determine access rights to local and remote resources.

Step:4 The user requests access to a mounted NetApp file system.

The storage system checks export options at the time the file system is mounted. Export options might affect the user’s ability to perform a requested action.

Step:5 The user requests access to data on the storage system that uses NTFS-style permissions, but the request contains UNIX-style UID and GIDs.

The storage system cannot use the UID and GIDs sent in the access request for the access check. Instead, the storage system must compare Windows user and group information to the file’s Window ACL to determine access for the user. Therefore, the storage system maps the UNIX user to the corresponding Windows user.

Step:6 The system queries the configured UNIX identity store, supplying the user’s UID, and requests the user name.

When the user requests access from a UNIX host, the request contains the user’s UID but does not contain the user name. After mapping is done by user name, the system must determine the UNIX user name before the mapping process can proceed.

Step:7 The UNIX name service returns the name of the UNIX user to the storage system.

Step:8 The storage system uses the UNIX user name to begin the user mapping process.
The storage system checks /etc/usermap.cfg and the LDAP user mapping entries, if configured, to see if there is a specific mapping entry for the UNIX user.

If there is a specific mapping entry, the storage system uses this entry for the user mapping process.

If there is not a specific mapping entry, the storage system assumes that the Windows user name is the same as the UNIX user name and automatically uses this name during the mapping process.

Step:9 The storage system queries Active Directory to determine if the mapped user name is a valid Windows user.

If the user name is a valid Windows user, the mapping process continues.

If the user name is not a valid Windows user, the user is either mapped to the generic NT user or access is denied, depending on how the wafl option wafl.default_nt_user is configured.

Step:10 Active Directory replies to the query, either supplying information on the user or returning an error indicating that the user was not found.

Step:11 The storage system queries Active Directory for the mapped Windows user’s SIDs (user SID and all group SIDs).

Step:12 Active Directory replies to the query, supplying information on the user, including the user’s SIDs.

Step:13 The storage system uses the user’s SIDs and compares them to the ACLs of files and directories for which access has been requested.

The system then determines whether the desired action is allowed.

Step:14 The system replies to the access request, either permitting the requested action if the file permissions allow it or denying the action if the permissions do not allow it.

MultiProtocol on NetApp storage system - Part 1

2013-05-15T23:09:00.001-04:00

Ever wonder how to allocate same volume to both Windows and Unix environment on NetApp systems? Many of my colleagues and friends have questions on this and so I determined to write up a article on this, hope it makes good sense.

After the drive is mapped or an export is mounted users that request data access on the network must be authenticated by whatever method is configured, typically windows clients are authenticated in a windows domain or very less frequently by the workstation. Unix users are authenticated by system or kerberos (network-based authentication).

Unix: NFS being connection less protocol, after the export is mounted and to request access of data a user needs to be authenticated to determine the user ID (UID) and group IDs (GIDs) and secondar/auxiliary GID's when user first logs in. This mapping information is retrieved from /etc/passwd and /etc/groups, network information system (NIS), or LDAP identity store.

Windows: CIFS being a session based protocol, the identity of the user can be determined just once when the session is first setup by mapping a drive. Data ONTAP always maps the user's windows identity to the user's UNIX identity during a CIFS session setup process.

Data ONTAP does not map the user's UNIX identity to the users windows identity during the UNIX/NFS authentication process. It only maps the user's UNIX identity to the user's Windows identity when a UNIX user tried to access files/folders with an effective windows NT ACL.

Points to Note:
==========
1. UNIX export rules are host-based; Windows share permissions and NTFS Windows ACLs are both user based.

2. CIFS share ACLs can be managed through the Windows Computer Management snap-in or through the cifs access command.

3. Export options are set on the storage system with the /etc/exports file. Export options apply to the host; however, the effective permissions for the user are based on the combined export and file access controls.

Data ONTAP Security Styles:
====================
Every volume and qtree is given a default security style at creation time. The default security style is set with the option options wafl.default_security_style [unix | ntfs | mixed] (default: unix).

Note: NEVER USE MIXED-MODE unless you determined to do so!

UNIX security style. For the UNIX security style, authorization to access directories and files is based on access allowed to the user’s UNIX UID and GIDs, with access rules following the UNIX style of file permissions through mode bits (rwxrwxrwx) or through NFSv4 ACL.

Windows security style. For the Windows security style, authorization to access folders and files is based on access allowed to the user’s Windows security identifier (SID) and Windows group SIDs, with access rules based on NTFS Windows security descriptors.

UserMapping:
==========
User mapping between a Windows user and a UNIX user is a fundamental part of NetApp multiprotocol access. Multiprotocol access depends on user mapping between a user’s UNIX identity and Windows identity to evaluate the user’s rights to perform file and folder operations within volumes and qtrees.

UNIX:
=====
If a user accesses data in a UNIX volume through CIFS, Data ONTAP first maps the Windows user name to its corresponding UNIX UID. If there is no corresponding UNIX user, by default the Windows user is mapped to the default UNIX user. The default user is designated with the following storage system option: options wafl.default_unix_user user_name (default: pcuser)

The designated default UNIX user may be any valid UNIX user designated by the storage administrator, but the default user must exist in the storage system’s /etc/passwd file, the NIS database, or the LDAP database. If this option is set to the null string, Windows users who are not mapped to a UNIX user are not allowed to log in. Therefore, if this option is set to the null string, it is vital that each Windows user can map successfully to a specific UNIX user.

NTFS:
====
If a CIFS user accesses NTFS security style files or folders with Windows ACLs, the SIDs of the CIFS user and groups are used to determine NTFS access rights. The user and group SIDs are compared to the file’s Windows security descriptor.

If a UNIX user accesses files and folders with Windows ACLs in an NTFS qtree, Data ONTAP maps the UNIX user to the Windows user based on the user mapping policy. Upon failed identity mapping, Data ONTAP rejects the access request. Upon successful mapping, Data ONTAP grants access based on the user’s mapped Windows user. The user’s Windows user SID and Windows group SIDs are used to determine NTFS access rights.

By default, if there is no corresponding Windows user, access is denied to the NTFS qtree from a UNIX host. This is because the option that is used to configure a generic Windows account is, by default, set to null.

wafl.default_nt_user is a storage system option that can be used to allow mapping of Windows users with no corresponding UNIX account to a generic Windows account. The default for this option is null. To enable access from a default user, add a valid Windows account to this option:

options wafl.default_nt_user corp\ntuser

MultiProtocol Access and Group Mapping:
==============================
Data ONTAP user mapping maps user names. It does not map groups. Due to the fundamentally different way that Windows groups and UNIX groups are configured, it is not possible to perform a consistent, exact, and simple mapping between UNIX groups and Windows groups. However, because group membership is critically important when determining file access, as part of the mapping process the mapped user’s group membership is retrieved and cached along with the user mapping information.

WAFL CREDENTIAL CACHE:
=======================
After performing UNIX-to-Windows user mapping, the results, including group membership for both the UNIX user and the Windows user, are stored in the storage system’s WAFL credential cache (wcc).

Windows-user-to-UNIX-user mapping is not stored in the wcc. Instead, with Windows-to-UNIX-user mapping, the UNIX user information is kept as part of the CIFS session credential. A fresh Windows-to-UNIX user mapping is needed only when a new CIFS session is established for that user

By default, the information is cached for 20 minutes. The cache time can be increased as follows:

options wafl.wcc_minutes_valid minutes (Valid range is 1 through 20,160.)

Please click here for more on this

Open Replicator Hot Pull Migration Simplified!

2013-05-13T17:52:00.000-04:00

Scenario:

=======

Microsoft Cluster Service (MSCS) with two Nodes:

======================================

*have powerpath latest version installed.

Step:1 Create a table similar to the one shown above and fill that information

Remote (source):
----------------------
Device Information: inq –sym_wwn
FA’s Mapped: symdev list –dev 1A0 –sid 1000 –multiport
FA’ WWN: symcfg list –FA all –sid 1000

Contol (target):
---------------------
1. Create a new device of same or greater size; map it to the FA’s.
2. Mask the newly create devices to the Control FA’s, this comes after zoning

Step:2 Zoning

For hotpull, all ports on control array that the target devices are mapped to must have access to the remote devices

In this case, zone as below:

Step:3 Mask the newly created device on target array to the source FA’s

Symmask –sid 2000 –wwn 5009035BAAA9BFE0 –dir 2A –p 0 add dev 1B0
Symmask –sid 2000 –wwn 5009035BAAA9BFE1 –dir 15A –p 1 add dev 1B0
Symmask –sid 2000 refresh

Step:4 Caution! Zone the host (windows cluster MSCS) to the control (target) symmetrix. DO NOT MASK new storage to the host. You will do this in step 9!

Which means the host still sees the old storage, we will let the host see new storage after the cutover process.

Step:5 Verify control and remote FA’s have visibility

Symsan list –sid 2000 –sanports –dir 2A –p0
Symsan list –sid 2000 –sanports –dir 15A –p1

Step:6 Verify if WWN’s of the remote devices are visible on the target FA’s

Symsan list –sid 2000 –sanluns –wwn 5009035BAAA9BFE0 –dir 2A –p 0
Symsan list –sid 2000 –sanluns –wwn 5009035BAAA9BFE1 –dir 15A –p 1

These commands should output the LUN WWN of the source device

Note: The device records might show incomplete due to SCSI reservations placed upon the devices by the cluster software

Step:7 Prepare the host for migration

1. Prior to begin the migration, the applications running on the cluster should be shutdown as the control (target) symm has no control on source IO activity, which might lead to data inconsistencies.
2. Work with windows admin to make sure the cluster resources are taken offline with at-most caution to move all resources to one node. In this instance I consider win_node2
3. Shut down the servers.

Step:8 Unassign the remote (source) devices from all cluster node HBA’s

Symmask –sid 1000 –wwn 100000000c9535011 –dir 1A –p 0 remove dev 1A0
Symmask –sid 1000 –wwn 100000000c9535012 –dir 16A –p 1 remove dev 1A0
Symmask –sid 1000 refresh

Note: This will prevent duplicate disks being visible to the cluster nodes when they are brought back online

Step:9 Mask cluster node HBA’s to the control (target) devices

Symmask –sid 2000 –wwn 100000000c9535011 –dir 2A –p 0 add dev 1A0
Symmask –sid 2000 –wwn 100000000c9535012 –dir 15A –p 1 add dev 1A0
Symmask –sid 2000 refresh

Step:10 Create device file that contains target (control) devices and remote (source) devices

hippo@unix ~]$ vi pair.txt

symdev=000180202000:1B0 60060160058714000090DE98F7DBDC11

Step:11 Create open replicator copy session

hippo@unix ~]$ symrcopy –f pair.txt create –copy –hut –pull –donor_update

-copy: Starts data transfer when the session is activated
-hot: Target (control) devices are online to the host during data transfer
-pull: Data being copied from remote array to the control array
-donor_update: All writes to the control device from the host will also be copied to the remote device. This protects against potential data loss that can be caused by failure of OR session during migration process.

Step:12 Query the session that all pairs are successfully created

hippo@unix ~]$ symrcopy –f pair.txt query

Check if status column says ‘created’

Step:13 Activate the Open Replicator session

hippo@unix ~]$ symrcopy –f pair.txt activate
hippo@unix ~]$ symrcopy –f pair.txt activate

Check if status column says ‘CopyInProg’

Step:14 Restart the cluster

1. Work with server team to power on the cluster nose that owned all the resources prior to the cluster being shutdown.
2. Confirm if the node sees all the devices, than restart the cluster service on the node.
3. Bring the cluster resources back online
4. Power on the other node in the cluster as well

Step:15 Monitor the open replicator session to ensure all the data is transferred successfully.

hippo@unix ~]$ symrcopy –f pair.txt query

Check if status column says ‘Copied’

Step:16 Terminate the open replicator session once the query shows data 100% copied for all the devices.

hippo@unix ~]$ symrcopy –f pair.txt terminate
hippo@unix ~]$ symrcopy –f pair.txt query

Done!

File-Based Migration using EMCopy

2013-05-12T15:33:00.000-04:00

Overview:
=======
1. EMCopy is windows based tool which can be used for migrating the shares from windows file server to celerra CIFS server
2. It lets you copy a file or directory (and included subdirectories) from and to an NTFS partition, keeping security the same on the copy as on the original.
3. EMCOPY allows you to back up the file and directory security—ACLs, owner information, and audit information—from a source directory to a destination directory without copying the file data.

Download EMCopy.exe:
==================
1. You need to contact your SAM or TC to have a copy of the emcopy.exe
2. I have a copy that I use --> send a email to hippodata@gmail.com

Install:
=====
1. Download to any location on your local computer, in this example I did it to "C:\emcopy"
2. Open command prompt, change to the directory

cd /d C:\emcopy

3. Run the command to see available options

C:\emcopy> emcopy.exe

Requirements:
===========
1. The migration User account should have Domain Admin membership and must be logged onto the system that is conducting the actual migration of data and attributes.
2. Make sure there are no unresolvable SID's on the source
3. Proper Group Membership and User Rights for the Windows account used for the migration.

Example: 1
========
Copy one directory with all its contents and its owner to a different directory on a Data Mover.

C:\emcopy>emcopy.exe c:\source\ h:\destination /o /s

Note: By default, the DACLs are copied when the /NOSEC option is not set. The owner attribute of the directories and files is also duplicated.

Example: 2
========
Copy source directory to destination directory including the local groups security entries, use this command syntax:

C:\emcopy>emcopy.exe c:\source\ h:\destination /o /s /lg

Note: You must use LGDUP before using EMCOPY with the /lg option.

Example: 3
========
C:\emcopy>emcopy.exe \\source\share\directory \\destination\share /o /s /c /r:3 /w:5 /log:"c:\log file.txt"

Note: 1. The "/o" option enables the copy of a file's owner. Without this option, the account used for the copy will be the owner of the copied files and directories.

2. The "/s" option enables the copy of the subdirectories.

3. The "/c" switch allows the process to continue after the retries.

4. The "/r:n" option specifies the maximum number of retries. By default it is set up
to 100.

5. The "/w:n" option specifies the time in seconds to wait between two retries.
By default it is set up to 30 seconds.

6. The "/log:filename" option allows to redirect the console messages to a new file.

Note: You can use /purge option as it enables the deletion of the files and directories fromt he destination tree which do not exist in the source one.

Post-Migration Activities:
===================
1. Analyse the emcopy log files that have been generated and run ‘Beyond Compare’ on source server to compare the directories copied

Download Beyond Compare link: http://www.filehippo.com/download_beyond_compare/

2. Right- click > Properties on the source and destination folders to check that windows can see the same amount of files and folders.

3. Check NTFS permissions at each share have been brought across by EMCopy

Useful EMC Primus:
===============
1. emc245596 - "EMCOPY 4.0x shows multiple errors like 'Trying to DELETE due to copy ABORT' and 'DELETED due to copy ABORT'"
2. emc288299 - "EMCOPY 4.xx shows error "DELETION FAILED""
3. emc224742 - "Warning EMCOPY and /z /d and /de command line switches"
4. emc135845 - "EMCopy data migration replaces UNIX primary group permission with default Windows Domain Users group when the /o switch is used in the EMCopy command."
5. emc195882 - "Failure using EMCopy 3.01: 'Error (1203): No network provider accepted the given network path.'"
6. emc46968 - "EMCopy utility fails with error 'Security Descriptor Error: Unable to set SD' while migrating data from source to target server."
7. emc254769 - "'Unable to set access time to' when copying files with EMCOPY"
8. emc70946 - "Emcopy fails to work as designed when using /purge command."
9. emc121242 -"EMCOPY fails to copy all files to destination when there is a mixture of long filenames and short filenames"
10. emc64705 - "Celerra Best Practices recommends using EMCopy to backup the customer ACLs."

Perl practicals - Lab 1

2013-05-12T12:57:00.001-04:00

Learning perl by practicing

Step:1 Find Perl Interpreter

hippo-MacBook-Pro:~ hippo$ which perl
/usr/bin/perl

Step:2.a Implicit Execution

Every script starts with shebang:"#!" which is not read as a comment. First line is also a place where you put your interpreter which in this case is perl.

#!/usr/bin/perl print "Perl Programming\n";

Make Perl Script Executable: chmod +x perl_script.pl

hippo-MacBook-Pro:~ hippo$ chmod +x perl_script.pl
hippo-MacBook-Pro:~ hippo$ ./perl_script.pl
Perl Programming

Step:2.b Explicit Execution

print "Perl Programming\n";

Make Perl Script Executable: chmod +x perl_script.pl

hippo-MacBook-Pro:~ hippo$ chmod +x perl_script.pl
hippo-MacBook-Pro:~ hippo$ perl perl_script.pl
Perl Programming

Step:3 Simple Perl script

#!/usr/bin/perl # print "Perl Programming Tutorial\n";

hippo-MacBook-Pro:~ hippo$ ./simple_perl_script.pl
Perl Programming Tutorial

Step:4 Current path to Perl modules

List all available current paths to perl modules:

perl -e 'print "@INC" . "\n";'

hippo-MacBook-Pro:~ hippo$ perl -e 'print "@INC" . "\n";'
/Library/Perl/5.12/darwin-thread-multi-2level /Library/Perl/5.12 /Network/Library/Perl/5.12/darwin-thread-multi-2level /Network/Library/Perl/5.12 /Library/Perl/Updates/5.12.4 /System/Library/Perl/5.12/darwin-thread-multi-2level /System/Library/Perl/5.12 /System/Library/Perl/Extras/5.12/darwin-thread-multi-2level /System/Library/Perl/Extras/5.12

Step:5 Variables

$ - Scalar Variable
% - Hash Variable
@ - Array
& - Subroutines

Step:6 Using Perl default variable $_

#!/usr/bin/perl
$_ = "Perl Programming default variable.\n";
print;

hippo-MacBook-Pro:~ hippo$ ./perl_default_variable.pl
Perl Programming default variable.

Step:7 Defined Function

#!/usr/bin/perl

# declare perl scalar do but not define value
$perl_scalar;
#we can use conditional operator '?:' to test perl defined funtion
$variable = defined($perl_scalar) ? "Variable \$perl_scalar is Defined!"
: "Variable \$perl_scalar is NOT Defined!";
print $variable."\n";

# declare perl scalar with value
$perl_scalar="perl";
$variable = defined($perl_scalar) ? "Variable \$perl_scalar is Defined!"
: "Variable \$perl_scalar is NOT Defined!";
print $variable."\n";

hippo-MacBook-Pro:~ hippo$ ./defined.pl
Variable $perl_scalar is NOT Defined!
Variable $perl_scalar is Defined!

Step:8 Scalar variable

#!/usr/bin/perl
#Scalars hold just single data type: string, number or perl reference
#Scalars definition in Perl
$scalar_number = -5;
$scalar_string1 = "In PERL Scalars are always referenced with \x24 in front of each variable name. ";
$scalar_string2 = "5 items";
#Undescore can be use for big numbers
$scalar_milion = 1_000_000;
#Print scalar values
print $scalar_number."\n";
print $scalar_string1."\n";
print $scalar_string2."\n";
print $scalar_milion."\n";
#perl scalar addition
print $scalar_number + $scalar_milion."\n";

hippo-MacBook-Pro:~ hippo$ ./perl_scalars.pl
-5
In PERL Scalars are always referenced with $ in front of each variable name.
5 items
1000000
999995

Step:9 Single-Quoted Strings

#!/usr/bin/perl

#Single-Quoted scalar strings
$scalar_string1='perl';
print "String 1: ".$scalar_string1."\n";
$scalar_string2='#!/usr/bin/perl';
print "String 2: ".$scalar_string2."\n";
$scalar_string3='Perl
Programming
Tutorial';
print "String 3: ".$scalar_string3."\n";
$scalar_string4='Perl\n';
print "String 4: ".$scalar_string4."\n";
$scalar_string5='\'\'\\';
print "String 5: ".$scalar_string5."\n";
$scalar_string6='';
print "String 6: ".$scalar_string6."\n";
$scalar_string7='I\'m reading Perl Programming Tutorial';
print "String 7: ".$scalar_string7."\n";

hippo-MacBook-Pro:~ hippo$ ./perl_default_variable.pl
String 1: perl
String 2: #!/usr/bin/perl
String 3: Perl
Programming
Tutorial
String 4: Perl\n
String 5: ''\
String 6:
String 7: I'm reading Perl Programming Tutorial

Step:10 Double-Quoted Strings

#!/usr/bin/perl
#Double-Quoted scalar strings
$scalar_string1="perl";
print "String 1: ".$scalar_string1."\n";
$scalar_string2="#!/usr/bin/perl";
print "String 2: ".$scalar_string2."\n";
$scalar_string3="Perl
Programming
Tutorial";
print "String 3: ".$scalar_string3."\n";
$scalar_string4="Perl\n";
print "String 4: ".$scalar_string4."\n";
$scalar_string5="\'\'\\\"";
print "String 5: ".$scalar_string5."\n";
$scalar_string6="";
print "String 6: ".$scalar_string6."\n";
# add "!" ASCII character in octal form !=041
$scalar_string7="I\'m reading Perl Programming Tutorial \041";
print "String 7: ".$scalar_string7."\n";
# add "@" ASCII character in hexadecimal form @=40

hippo-MacBook-Pro:~ hippo$ ./perl_strings.pl
String 1: perl
String 2: #!/usr/bin/perl
String 3: Perl
Programming
Tutorial
String 4: Perl

String 5: ''\"
String 6:
String 7: I'm reading Perl Programming Tutorial !

Step:11 String Operators

#!/usr/bin/perl

#Scalar string Operators
$scalar_string1="pe"."rl";
print "String 1: ".$scalar_string1."\n";
$scalar_string2="Perl Programming Tutorial " x (1+1);
print "String 2: ".$scalar_string2."\n";
$scalar_string3="3"."\ttabs" x 3;
print "String 3: ".$scalar_string3."\n";
$scalar_string4="Perl\x20".'Programming '."Tutorial";
print "String 4: ".$scalar_string4."\n";
$scalar_string5=9x5;
print "String 5: ".$scalar_string5."\n";

hippo-MacBook-Pro:~ hippo$ ./perl_strings.pl
String 1: perl
String 2: Perl Programming Tutorial Perl Programming Tutorial
String 3: 3 tabs tabs tabs
String 4: Perl Programming Tutorial
String 5: 99999

Step:12 Non-Decimal Integers

#!/usr/bin/perl

#perl binary integer
$hash_binary_integer = 0b10000;
#perl octal integer
$hash_octal_integer = 020;
#perl hexadecimal integer
$hash_hexadecimal_integer1 = 0x10;
$hash_hexadecimal_integer2 = 0x124c_78_aa;

print $hash_octal_integer."\n";
print $hash_binary_integer."\n";
print $hash_hexadecimal_integer1."\n";
print $hash_hexadecimal_integer2."\n";

hippo-MacBook-Pro:~ hippo$ ./non_decimal_integer.pl
16
16
16
307001514

Step:13 Scalar Constant Variable

#!/usr/bin/perl

$ordinary_scalar = 5;
$ordinary_scalar = 10;

print $ordinary_scalar."\n";

#perl constant declaration
*SCALAR_CONSTANT = 5;
$SCALAR_CONSTANT = 10;

hippo-MacBook-Pro:~ hippo$ ./perl_constant.pl
10
Modification of a read-only value attempted at ./perl_constant.pl line 10.

Step:14 String And Numeric comparison Operators

#!/usr/bin/perl
# String comparison
if ( 'Perl' eq 'perl' ) {
print "TRUE\n";
} else {
print "FALSE\n";
}

# Numeric comparison
if ( '2.4' != '2.6' ) {
print "TRUE\n";
} else {
print "FALSE\n";
}

hippo-MacBook-Pro:~ hippo$ ./comparision.pl
FALSE
TRUE

Perl Scripting - Part 3

2013-05-12T12:07:00.002-04:00

Please click here for part-2

Subroutines:
=========
-Like any good programming langauge Perl allows the user to define their own functions, called subroutines.
-They may be placed anywhere in your program but it's probably best to put them all at the beginning or all at the end.

A subroutine has the form:

sub mysubroutine
{
print "Not a very interesting routine\n";
print "This does the same thing every time\n";
}

-Regardless of any parameters that we may want to pass to it.
-All of the following will work to call this subroutine.

Notice that a subroutine is called with an “&” character in front of the name:

&mysubroutine; # Call the subroutine

&mysubroutine($_); # Call it with a parameter

&mysubroutine(1+2, $_); # Call it with two parameters

Parameters:
=========
-In the above case the parameters are acceptable but ignored.
-When the subroutine is called, any parameters are passed as a list in the special @_ list array variable.
-This variable has absolutely nothing to do with the $_ scalar variable.
-The following subroutine merely prints out the list that it was called with. It is followed by a couple of examples of its use.

sub printargs
{
print "@_\n";
}
&printargs("perly", "king"); # Example prints "perly king"
&printargs("frog", "and", "toad"); # Prints "frog and toad"

-Just like any other list array, the individual elements of @_ can be accessed with the square bracket notation:

sub printfirsttwo
{
print "Your first argument was $_[0]\n";
print "and $_[1] was your second\n";
}

-Again it should be stressed that the indexed scalars $_[0] and $_[1] and so on have nothing to with the scalar $_ which can also be used without fear of a clash.

Returning Values:
=============
-Result of a subroutine is always the last thing evaluated.
-This subroutine returns the maximum of two input parameters.

An example of its use follows.

sub maximum
{
if ($_[0] > $_[1])
{
$_[0];
}
else
{
$_[1];
}
}

$biggest = &maximum(37, 24); # Now $biggest is 37

-The &printfirsttwo subroutine above also returns a value, in this case 1. This is because the last thing that subroutine did was a print statement and the result of a successful print statement is always 1.

Local Variables:
============
-The @_ variable is local to the current subroutine, and so of course are $_[0], $_[1], $_[2], and so on.

-Other variables can be made local too, and this is useful if we want to start altering the input parameters.

-The following subroutine tests to see if one string is inside another, spaces not withstanding.

An example follows.

sub inside
{

local($a, $b); # Make local variables
($a, $b) = ($_[0], $_[1]); # Assign values
$a =~ s/ //g; # Strip spaces from
$b =~ s/ //g; # local variables
$a = $b;
$b = $a; = What result they will be?
# $a = $_[1];
# $b = $_[0];

print "the first element is: $a\n";
print "the second element is: $b\n";

}

&inside("lemon", "dole money");

Please click here for perl practicals - Lab1

Perl - Introduction

2013-05-12T11:57:00.001-04:00

Ever wonder what's the starting point for learning any scripting for automating your environment? Well, this can be your place to start learning industry's one of the most powerful scripting tool 'perl'. Make sure you follow all the related posts and practice more.

Overview:
========
-Perl fits in between the C/C++ programming languages and shell level utilities like awk, grep and sed.
-Perl beats shell programming languages in speed and beats C/C++ in ease of use.
-Perl is quite high level with “dense” and portable codes with the following attributes:
-Faster to write and read
-Faster to debug
-Easy to maintain.

Perl Popularity:
============
-Just like many popular tools, the creator of perl released it to the Usenet community.
-Users of the early perl gave feedback and made requests to the creator.
-Perl grew in features as well as popularity.
-Perl was initially on Unix systems, and it is now available on nearly all systems.

How to run a simple perl program?
==========================
-A perl program can be invoked by either execute it directly
-Or pass the source code to the perl interpreter.
-The first two characters in a shell program “#!” tells the computer what to use to execute the rest of the program.
-Perl is typically installed:
/usr/bin/perl or /usr/local/bin/perl.
-If not, you need to find out and specify it in the first line of your perl code.

Snippet:
----------
unix>
unix> cat hello.pl
#!/usr/bin/perl
print "Hello World!\n";

unix> ls -l hello.pl
-rw-r--r-- 1 hippo users 4 May 8 2:52 hello.pl

unix> hello.pl
hello.pl: Permission denied.

unix> perl hello.pl
Hello World!

unix> chmod +x hello.pl
unix> ls -l hello.pl
-rwxr-xr-x 1 sun users 4 May 12 2:52 hello.pl

unix> hello.pl
Hello World!
unix>

Turn on warning:
=============
-By default, perl does not warn you of suspicious coding.
-To run a perl program with warning on, use the one of the two forms:

% perl –w perl_prog

#!/usr/bin/perl –w
require 5.001;

-The second line indicates which version of perl the program expects.

Help:
====
-Some websites for you to browse:
http://www.cpan.org
http://www.perl.com
-You can simply google for any questions you may have, and you have a good chance of getting your answers.

Please click here for Part-1

Perl Scripting: Part 2

2013-05-12T11:42:00.003-04:00

(Please click here for part - 1)

Key Concepts:
==========
Associate Arrays (Hash Arrays)
File Input and output
Strings
Subroutines
Running external programs

Hash arrays:
=========
-Hash arrays are also called associate arrays.
-Instead of using “@” and “[ ]”, “%” and “{ }” are used for hash arrays.
-Hash arrays are used to find the value based on a key. A key is used to retrieve the value associated with key.
-To access an element from an hash array:
$h_array{key}
-Perl borrowed associate array from awk.
-Hashing helps to speed up searching significantly specially for large amount of data.

Hashing and hash function:
====================
-Hashing is a method for directly referencing records in a table by doing arithmetic transformations on keys into table addresses.
-If keys are distinct integers from 1 to N, then the records can be stored in the table by the key position.
-In this case, the data can be retrieved directly from the table using the key.
-The first step in using hashing is to transform the search key into a table address.
-The function or method used to convert the key into table address is the hash function.
-Ideal hash functions should map different keys into different table addresses.
-If different keys hash into the same address, then hash collision happens.
Good hashing functions produce fewer collisions.

Array and Hash comparison:
=====================
Access Data From Array:
-Start from index 0
-for each record, compare the key to that of the record.
-If match then access the data: retrieve or update.

index data
----- ----
0 record0
1 record1
2 record2
3 record3
4 record4
5 record5

Access Data From Hash:
-convert the key to hash table index.
-Get the first record from the table index.
-Search thru the list until found.

hashtable
---------
0
1 --------> record1a
2
3
4 --------> record4a ------> record4b
5

Hash Function:
===========
-Most of the time, the search key is a string.
-When converting a string into a hash table index, the following factors must be considered:
a. Speed of the function
b. Index less than zero
c. Index larger than table size
d. Frequency of collision
e. Since the names used for hash typically do not have limits, complex computation for a hash function may be too expensive.
f. The goal of the hash function is to minimize collision, but in practice, collisions always happen.

Closed Hash:
===========
-Closed hashing is used when the number of entries is known and it is less than the size of the hash table.
-Closed hash is also referred to as open-addressing (address, not hash. Do not let the terms confuse you).
-The simplest close hash (open addressing) is called linear probing.
-The way close hash works (for searching):
a. Use hash function to get the table index.
b. If the entry at the table index is the one then search is successful.
c. If the entry at the table index is empty, then search not found.
d. If the entry at the table index is not what is being searched, continue to search the next address.
e. Continue to search until an empty entry, a match or end of the table.

Open Hash:
==========
-Open hashing is also called separate chaining.
-When multiple entries hash to the same table index, a linked list is built to keep all the entries.
-Under open hash, the search within the table is eliminated.
-If the table size is H, entries will be distributed evenly to the H linked lists if the has function is good.
-Compare a hash table with size H and a single linked list, the search using a hash table can be close to H times faster (not exactly due to the hash function overhead and the imperfection of the hash function).

Hash array access: existence
====================
The function exists can be used to check the existence of a given key:

if (exists $books{“cats”}) { print “Yes, there is an entry for cats!\n”; }

Hash array access: assign/overwrite
===========================
-If an entry does not exist, it can be assigned: $name{“john”} = “tall guy”;
-If an entry exists already, it will be overwritten.
-A hash array can be copied to another one:
%new_hash = %old_hash; %inverse_hash = reverse %old_hash;
-A hash array can also be assigned name/value pairs directly:
%name_hash = (“key1”, 12, “key2”, 13, “key3”, 14);

Hash array access : name/value pair (=>)
==============================
-When assign name and value pairs to a hash array, it is difficult to tell which entry is key and which is value.
-Perl provides an easy way to represent name value pairs:
%names = ( “key1” => “value1”, “key2”=> “value2” )
-This way, it is very clear which entry is the name and which entry is the value.
-The big arrow (=>) can be used to replace a comma. It is also called fat comma.

Hash array access: delete
===================
-The delete function can be used to remove a given key.
-If there is no such key to be deleted, there will not be any warning or error.
Syntax: delete $h_array{“key”} ;
-It is not he same as storing undef into that hash element.
-The function exist returns true if a key value is “undef”.
-The function exist returns false if a key is deleted.

Hash array access: keys and values
==========================
-Hash array supports two functions:
a. keys: get the entire keys at once
b. values: get the entire values at once.

-Both keys and values return empty lists if no elements exist.

Snippet:
----------
unix>cat keys_values.pl
#!/usr/bin/perl -w
%hash = ("a" => 1, "b" => 2, "z"=>4, "x" =>23);

@ks = keys %hash;
@vs = values %hash;

printf "keys:";
printf " %s " x @ks, @ks ;
printf "\n";
printf "values:";
printf " %s " x @vs, @vs ;
printf "\n";
snpsemt225:yinglir>keys_values.pl
keys: a b x z
values: 1 2 23 4
unix>

File Handles:
=========
-Variables that represent files are called file handles.
-File handles do not have any special character ($, @, &).
-They are typically created as UPPER case variables.
-All file handles in perl are global. They can not be allocated locally.

The standard files:
=============
-Before a perl program runs, three standard files are opened:
STDIN
STDOUT
STDERR
-The < > operator returns one line from standard input. It returns undef when there is no more inputs.

$line = <STDIN>
while ($line = <STDIN>) {
chomp ($line);
}

Open and Close files:
===============
-The “open” and “close” operators work similar to other programming languages:
open (F1, “filename”); # open “filename” for read with handle F1.
open(F2, “>filename”); # open “filename” for writing as file handle F2
open(F3, “>>filename”); # open “filename” for appending .
close (F1); # close a file handle.

-Open can be used to establish read/write connection to a separate process launched by the OS (on Unix):
open (F, “ls –l |”); # open a pipe to read from an ls process
open (F, “| mail $addr”); # open a pipe to write to a mail process.

-A convenient way to exit a program: open(F, $filename) || die “could not open $filename\n” ;

Read data from a file:
================
-In a scalar context, the input operator reads one line at a time.
$line = <F> ; # reads in a line at a time.

-In an array context, the input operator reads the whole file into the memory as an array of its lines:
@a = <F> ; # reads the whole file

-The global variable “$/” which is the end-of-line marker (default to \n). Setting this to undef will cause the file to be read in as a single line:
$/ = undef;
$all_lines = <F> ; # read the whole file into one string.

Print output:
==========
By default, print uses STDOUT to send the outputs.
An output file handle can be specified to print: print F “here”, “ comes”, “ the rain!\n”;

Snippet:
----------
#!/usr/bin/perl -w
$fname = "fileline";
$line = "";
open ("here", $fname) or die ("Could not pen $fname\n");
while (<here>) {
chomp($_);
$line = $_;
print "$line\n";
}
close ("here");

Printing array:
===========
-Neither print or printf can print arrays directly.
-printf can be used with a little effort to dump an array content.
-Note that when array is used in the context of a scalar, it returns the number of elements.
-“format_str x @ary” can be used to create the format string.
-“x” is the repetition operator and @ary gives the number of elements.

Snippet:
----------
unix>cat print_array.pl
#!/usr/bin/perl -w
my @ary = qw/cat dog pig fish fly/ ;
my $format = "Array content: \n" . ("%15s \n" x @ary);
my $ary_n = “”;
printf STDOUT $format, @ary;

print "Using print:\n";
print @ary ;
print "\n";

print "Using printf:\n";
$ary_n=@ary;
printf “$ary_n\n” ;
print "\n";

unix>print_array.pl
Array content:
cat
dog
pig
fish
fly
Using print:
catdogpigfishfly
Using printf:
cat
unix>

String Processing (binding operator):
===========================
-String manipulation is one of Perl’s most powerful features.
-Perl utilizes regular expression extensively for string manipulation.
-Perl uses the binding operator (=~) to match pattern: ($string =~ /pattern/)
-The expression ( $string =~ /pattern/ ) returns true as long as the pattern exists in the string.
-($string =~ /pattern/i ) makes the matching case insensitive.
-If it matches, $1, $2 and etc will contain the tokens in the string. Note that this is equivalent to the \1 \2 in RegExp. It is supported by Perl as well.
-If the string matches, three special variables can be used:
a. $& (dollar-ampersand) holds the matched string
b. $` (dollar-back-quote) holds the string before the matched portion
c. $’ (dollar-quote) holds the string after the matched portion.

The m// construct:
=============
-The /pattern/ in the previous examples is actually a short cut for m/pattern/. The slash used as the delimiter, the “m” character can be omitted.
-With the “m” (for match) character, then any character can be used as the delimiter.
-The following are equivalent:
“string” =~ m/str/ ;
“string” =~ m”str” ;
“string” =~ m’str’ ;
“string” =~ m#str# ;

Limit over matching with ?
===================
-Both * and + tend to be greedy and over match (for the largest match).
-m/{(.*)}/ will match the largest string with “{}”.
-If the string is “{ group 1} and {group 2}”, m/{(.*?)}/ will return “group 1} and {group 2”.
-If ? (matches at most one occurrence) is used after * or +, Perl will match the shortest string instead of longest string.

String option modifiers: i, s, x, g
========================
-The “i” option makes the matching case insensitive.

Snippet:
----------
print “Do you want to continue?”
chomp ($_ = <STDIN>) ;
if (/yes/i) { # when matching to the default variable $_, $_ =~ can be omitted.
print “Thank you for the positive response!\n” ;
}

-The “s” option makes the (.) dot match to include newline.

Snippet:
-----------
$str = “The dog runs \n after the cat.\n” ;
if ($str =~ /dog.*cat/s) { # the matches fails w/o the “s”. print “See dog ahead of cat.\n” ;
}

-The “x” allows white space in the pattern to improve readability.
“/-?\d+\.?\d*/” and “/-? \d+ \.? \d* /x” are equivalent.

-The “g” allows the string substitution applied repeatedly in the string

-A “s” in front of the matching substring can be used to substitute the matching with a new substring:
$str =~ s/old/new/ig ; # i for mixed case, g for matching repeatedly.

-$1 and $2 can be used to refer to parts of the matched string (with groups):
$x = “This dress exacerbates the generic betrayal that is my legacy.\n” ;
$x =~ s/(r|l)(\w)/z$2/ig ; # r or l followed by a word char. ##
$x is now: “This dzess exacezbates the genezic betzayal that is my zegacy.

String “split”:
==========
-The split construct takes a regular expression and a string to return an array of all the substrings.
-If “-1” is passed as the third argument, an empty element is added to the end of the array.

Snippet:
-----------
split (/\s+/, “this is a string”); gives (“this”, “is”, “a”, “string”)

split (/\s*,\s*/, “that , tree, is,very ,tall”); gives (“that”, “tree”, “is”, “very”, “tall”)
split (/\s*,\s*/, “that , tree, is,very ,tall”, -1); gives (“that”, “tree”, “is”, “very”, “tall”, “”)

String “tr”:
========
-The “tr” construct can be used to convert characters in the string:
$str =~ tr/a/b/ ; $str =~ tr/A-Z/a-z/ ;

-Note that the two sets should match in sizes:

Snippet:
----------
unix>cat tr.pl

$str = "this is a cat and my cat can take on a dog.\n";
print $str ;
$str =~ tr/cat/dog/;
print $str ;

unix>
unix>perl tr.pl
this is a cat and my cat can take on a dog.
ghis is o dog ond my dog don goke on o dog.
unix>

Local Variables:
============
-Starting with Perl version 5, the “my” construct were introduced to create local variables:
my $var ; # declares a variable $var
my $v1 = “cat” ; # declares v1 and assign it with “cat”.
my @ary = (1..10) ; # declars array @ary and assign 1 thru 10 to it.
my ($x, $y); # declares two local variables $x and $y
my ($p, $q) = (100, 200) ;

-Local variables are mostly used in subroutines.

Return multiple values:
=================
-What if multiple values need to be returned to caller?
-One approach can be packing multiple values into an array and return it.
…
my ($str1, $value1, $str2) = myroutine();
…
sub myroutine {
…… @ary = (“this”, -1, “that”);
return (@ary);
}

Running External Programs:
======================
-Perl can invoke other programs using the “system” function.
-Perl passes the argument to “system” as a command line to the operating system.
-System returns 0 when the program successfully completes and the global variable “$?” should be set to an error.

system (“mail hippo\@rkguru.com < mail.txt”) == 0 | die “system error $?”;

Please click here for part - 3