Nerdy Notes

Manga Reading Sites - Digital and Physical

2022-05-28T07:49:00.003-04:00

Subscriptions

Physical

Digital

Media Tracking and Recommendation Services

2021-09-25T13:39:00.014-04:00

Anime

Movies

https://www.imdb.com

TV Shows

https://www.tvtime.com

Media Streaming Service Locators

2021-09-25T13:39:00.011-04:00

Anime

TV and Movies

https://www.justwatch.com/ca

Windows 10 Hardware Temperature Monitoring Tools

2020-09-27T16:30:00.001-04:00

Tool to monitor temperature of hardware components in Windows 10

Open Hardware Monitor - portable open source version of hwmonitor

Core Temp - more CPU measurement features

hwinfo - similar to Open Hardware Monitor and has thermal throttling detection

Terminal Ping Sweep and Port Scan

2020-09-23T10:20:00.002-04:00

Result: List of IP addresses that respond to ping on the specified prefix. Scan common ports on devices that respond to ping.

Modify the prefix variable as necessary, and paste into terminal

prefix="192.168.0."

Copy and Paste the rest of the following script into the terminal application

for i in {1..100}
do
        ping -c1 -W1 -i0.2 $prefix$i > /dev/null
        if [ $? -eq 0 ]
        then
                echo $prefix$i
                nc -w 1 -vz $prefix$i 1-1056 2>&1 | grep succeeded
        fi
done

Windows cmd ping sweep

2020-08-15T10:52:00.001-04:00

Instead of installing a third party application which has the possibility to have spyware or other bad stuff in it. If you only need to do a ping sweep with no additional features it's probably easier just to do it in CMD.

Open up cmd
Run the following

for /L %z in (1,1,254) do @ping 10.0.0.%z -w 10 -n 1 | find "Reply"

Where:

(1,1,254) indicates to test every address from 1-254
10.0.0. is the network address prefix of addresses you want to test
%z is filled in with numbers from (1,1,254)
-w 10 - only waits 10ms before moving on
-n 1 - only tries once before moving on
find "Reply" looks for a succesful ping result and ignores unsuccessful

Clean Console Log Output

2015-12-07T13:35:00.000-05:00

To remove colour codes and redraw characters from console to make console logs more readable.



#!/usr/bin/env perl

while (<>) {

    s/ \e[ #%()*+\-.\/]. |

       (?:\e\[|\x9b) [ -?]* [@-~] | # CSI ... Cmd

       (?:\e\]|\x9d) .*? (?:\e\\|[\a\x9c]) | # OSC ... (ST|BEL)

       (?:\e[P^_]|[\x90\x9e\x9f]) .*? (?:\e\\|\x9c) | # (DCS|PM|APC) ... ST

       \e.|[\x80-\x9f] //xg;

    print;

}

Usage:



less /path/to/console.log | cleanLog | less

Limiting Bandwidth Using Robocopy

2015-07-21T19:30:00.000-04:00

Robocopy does not have any built in bandwidth limiter by rate specifically.

But what you can do is use the inner packet gap switch /ipg:n to simulate a lower bandwidth by adding more delay between packets.

The equation to use to calculate is as follows:

Where
D is the calculated latency or inner packet gap
Bd is the desired bandwidth you would like to transfer at
Ba is the available bandwidth to you

Example:
If my
Bd is 200kbps
Ba is 1000kbps



D=((1000-200)/(1000*200))*512*1000=2048

Therefore my /ipg:n will be /ipg:2048

References:
h ttps://yellowtriangle.wordpress.com/2012/06/28/bandwidth-throttling-with-robocopy/
http://www.zeda.nl/index.php/en/copy-files-on-slow-links
https://technet.microsoft.com/en-us/library/cc733145.aspx

Sourcing Files in Bash

2015-07-11T06:06:00.000-04:00

I never quite understood why when you changed .bashrc you had to type the source command in order to make changes active.

Now I understand and it's fairly straight forward.

Your current bash environment is defined by a PID that can be found using



echo $$

Any newly spawned bash processes for scripts will fork off of that process and be given a new process ID.

All variables for that newly spawned process will be local to that newly forked process.

If you would like the variables that would typically be local to your forked script to be active in your current shell you can use the source command in order to make them local to your current shell.

This acts in the same way when you would like to make variables in the .bashrc script local to your current shell. When the computer boots it runs .bashrc and puts variables in your current shell. If you would like to make any variable changes to re-read into your current shell. Similarly to how you would make variables to a remote script available to your local shell by using the source command you can do the same for .bashrc

Example:



source script.sh

The above will make all variables in script.sh available to your local shell at echo $$



source .bashrc

The above will make all variables in script .bashrc available to your local shell at echo $$

Create Redhat Syncronized Repo with Reposync

2015-04-29T08:06:00.000-04:00

Prerequisites

Server registered with the appropriate version of RedHat
HTTP/FTP server to host repo files

Instructions

Install required components for managing yum repositories and createrepo utility for building new repo indexes.



yum install yum-utils createrepo



rhn_register



subscription-manager register --username $username  --password $password --auto-attach

Repos will be populated automatically upon registration and you can then list them using:



subscription-manager repos --list

Download the repos to a path of your chosing



reposync --gpgcheck -l --repoid=$repoid --download_path=$path

Create the repo against the directory where your rpm packages exist



cd $path

createrepo $path

Enable the local repo



vi /etc/yum.repos.d/rhel.repo



[rhel-repo]

name=Red Hat Enterprise Linux 7 - x86_64

baseurl=http://$server/$path

enabled=1

gpgcheck=0

Ensure the repo works, this will likely build the index at the same time. I've seen elsewhere where cache is refreshed using yum makecache but I haven't run into a scenario where I have to do that yet.



yum repolist

References:
https://access.redhat.com/solutions/23016
http://kb.kristianreese.com/?View=entry&EntryID=77

Kickstart Drive Mapping

2015-02-17T09:45:00.001-05:00

Ordering of Storage Devices Messing up install

A problem that occurs commonly in kickstart is that kickstart will detect storage devices in a different order that which you intend the OS to install in.

This most commonly happens with Dell devices and virtual disks.

The Dell device will name its first virtual device /dev/sda which will cause installation problems when you try to specify that you want your kickstart installation to install on /dev/sda which is likely going to be one of Dell's virtual media devices.

A way to get around this is to create a kickstart script that looks at some criteria of physical devices first and then determines which drive to install to.

Finding Drive Mapping

http://www.redhat.com/archives/kickstart-list/2012-October/msg00014.html
https://access.redhat.com/solutions/273843
https://access.redhat.com/solutions/188463
https://access.redhat.com/solutions/58467
https://access.redhat.com/solutions/58561

Interactive Pre-Install to Test

https://stuffivelearned.org/doku.php?id=os:linux:centos:kspostscript
http://hintshop.ludvig.co.nz/show/interactive-pre-post-install-scripts-redhat-kickstart/

Example of My Partitioning Script



# Couldnt figure out how to make piping with while loop output work properly

# So I wrote to a temp file which seems to be the most compatible method of

# Performing while loop redirection in multiple shells

list-harddrives > /tmp/list-harddrives



# Loop to filter through attached HDDs and their sizes in MiB

# Will find the smallest disk larger than 15GB and set it as /

while read DISK DSIZE

do

DSIZE=$(echo "$DSIZE" | awk -F '.' '{print$1}')

if [ -z $rootDisk ]; then

rootDisk=$DISK

rootSize=$DSIZE

echo "Root Disk was unset and is now: $rootDisk"

fi



if [ $DSIZE -gt 143305 ]; then

if [ $DSIZE -lt $rootSize ]; then

rootDisk=$DISK

rootSize=$DSIZE

echo "Root Disk overriden due to smaller disk that meets requirements: $rootDisk"

fi

fi

done < /tmp/list-harddrives

# Echo result out to console for debug
echo "Root Disk: $rootDisk"

# Create another temp file to feed into the kickstart later
# It will choose the root disk that was previously figured out as the primary disk
cat << EOF >> /tmp/partitioning

# Specify the bootloader as required for RedHat automated installation (--location=mbr is default)

bootloader --driveorder=${rootDisk}

# Without the 'zerombr' option specified, uninitiated disks will require manual intervention

zerombr

# To clear the existing partitions, use the clearpart option and include the --all parameter

clearpart --all --initlabel



# Create /boot partition (100MB is not large enough anymore so 500MB is used)

part /boot --size=500 --ondisk=${rootDisk}

# Create a pv to be used with LVM, start with 100MB partition and grow to the remaining size that is left on disk

# This will also ensure that we use the root disk we determined earlier specifically

part pv.01 --size=100 --grow --ondisk=${rootDisk}



# Add the previous pv to the volume group

volgroup VolGrp pv.01



# Carve out space in the volume group for /, swap, and /d2 (data)



# Root partition will always be 10GB

logvol / --vgname=VolGrp --size=10000 --name=Vol1

# Swap will be what is recommended by RedHat for each Distro (no guarantees on which disk)

logvol swap --vgname=VolGrp --recommended --name=Vol2 --fstype=swap

# d2 is the a data partition which will use any remaining space (no guarantees on which disk)

logvol /d2 --vgname=VolGrp --size=100 --name=Vol3 --grow

EOF

Reference for include scripts and multi-drive kickstart partitioning:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/s1-kickstart2-preinstallconfig.html

ESXi 5.1 PXE Install

2015-02-02T11:11:00.001-05:00

ESXi PXE Install is very similar to any other kind of automated PXE deployment.

There are a few finicky components you'll have to deal with but overall it's fairly straight forward.

This is assuming you already have PXELINUX fully installed and configured along with DHCP and TFTP Server

Download the ISO
Mount the iso (I'm mounting in BSD)

mdconfig -a -t vnode -f isoimage.iso -u 1 mount -t cd9660 /dev/md1 /mnt/iso/

Create a directory to store the raw iso files

mkdir -p /images/esxi/5.1/1065491/dell-a03/

Rsync the iso contents over

rsync -av /mnt/iso/ /images/esxi/5.1/1065491/dell-a03/

unmount the iso

umount /mnt/iso mdconfig -d -u 1

Create an "isolinux.cfg" file which basically is just used to point to the files required to boot up and run the kernel/kickstart files. This file will exist typically under the pxelinux.cfg directory either as the file called "default" or will match the mac-address of the machine you would like to image

/images/kickstart/ESXI-5.1-1065491-dell-a03.ks DEFAULT install  LABEL install KERNEL images/esxi/5.1/1065491/dell-a03/mboot.c32 APPEND -c images/esxi/5.1/1065491/dell-a03/boot.cfg MENU LABEL Dell ESXi-5.1-Update 1-1065491(A03) ^Installer

Modify the boot.cfg file so that it does not reference the tftp root directory thanks to http://brain.serenity-networks.com/networking-computers/vmware/esxi-5-5-5-pxe-installation-via-syslinux

sed -e "s#/##g" -e "3s#^#prefix=/`basename $PWD`/\n#" -i.bak boot.cfg

Then you will modify the prefix parameter with the path to your files on tftp. In my case it would be "/images/esxi/5.1/1065491/dell-a03/"

 bootstate=0 title=Loading ESXi installer prefix=images/esxi/5.1/1065491/dell-a03/ kernel=tboot.b00 kernelopt=ks=ftp://ftp/images/kickstart/scripts/ESXI-5.1-1065491-dell-a03.ks modules=b.b00 --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- s.v00 --- misc_cni.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_tg3.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- net_bna.v00 --- scsi_bfa.v00 --- ima_be2i.v00 --- scsi_be2.v00 --- scsi_lpf.v00 --- net_igb.v00 --- net_ixgb.v00 --- scsi_mpt.v00 --- scsi_mti.v00 --- ima_qla4.v00 --- net_qlcn.v00 --- scsi_qla.v00 --- scsi_qla.v01 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- weaselin.t00 --- esx_dvfi.v00 --- xlibs.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- misc_dri.v00 --- net_be2n.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_nx_n.v00 --- net_qlge.v00 --- net_r816.v00 --- net_r816.v01 --- net_s2io.v00 --- net_sky2.v00 --- net_vmxn.v00 --- ohci_usb.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_rst.v00 --- uhci_usb.v00 --- tools.t00 --- dell_con.v00 --- xorg.v00 --- imgdb.tgz --- imgpayld.tgz build= updated=0

References:
Example kickstart config from yellow-bricks: http://www.yellow-bricks.com/2011/07/19/esxi-5-0-and-scripted-installs/
Kickstart valid parameters: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004582
Example kickstart from william lam: http://www.virtuallyghetto.com/2011/07/automating-esxi-5x-kickstart-tips.html
Blog to show how to structure boot.cfg: http://brain.serenity-networks.com/networking-computers/vmware/esxi-5-5-5-pxe-installation-via-syslinux/
Vmware documentation on PXE Install: https://pubs.vmware.com/vsphere-50/index.jsp#com.vmware.vsphere.install.doc_50/GUID-B9DB94CA-4857-458B-B6F1-6A688726AED0.html

Linux Out Of Memory Management (Killer)

2015-01-20T12:30:00.001-05:00

A summary of what an out of memory condition is is as follows:

Linux over allocates memory to processes VSZ giving them more than what they need.
It overcommits this memory allowing a larger VSZ than total memory availble
This generally isnt a problem since linux doesnt generally use up all the allocated memory
If processes actually require more memory than is available (due to their over allocation) it will kill "unimportant" processes to free memory and keep the system running

The messages for OOM killer can be found in:
/var/log/messages
Normally with the following keywords:
Out of Memory: Killed process

There are methods to mitigate the actions of OOM killer to prevent it from killing certain processes which can be found: http://unix.stackexchange.com/questions/136291/will-linux-start-killing-my-processes-without-asking-me-if-memory-gets-short/136294#136294 and here http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html

The OOM calculator calculates using the following method: http://unix.stackexchange.com/questions/153585/how-oom-killer-decides-which-process-to-kill-first

References:
http://en.wikipedia.org/wiki/Out_of_memory

Memory Management In Linux

2015-01-20T11:53:00.001-05:00

Using ps we can find out how much memory is allocated to a process (VSZ) and how much memory a process is actually using (RSS)



ps -e -o pid,vsz,rss,comm= | sort -n -k 2

The above will show you the process id, how much memory is allocated to a process, how much the process is actually using, and a comment of process description. It will then sort by how much memory is allocated to a process for you to see which apps are using up a lot of VSS.

Long running processes are useful to know as well when looking for memory hungry apps that have been running a long time



ps -ef --sort=start_time

References:
http://linuxconfig.org/ps-output-difference-between-vsz-vs-rss-memory-usage
http://serverfault.com/questions/27887/how-to-sort-ps-output-by-process-start-time
http://linuxcommando.blogspot.ca/2008/09/how-to-get-process-start-date-and-time.html

Windows Network Interface Priority

2014-11-26T14:07:00.002-05:00

If you have two default gateways it can be tricky to figure out which interface it will take to get out to the internet.

Windows will use the following criteria to choose which path to take:

Lowest Metric - Calculated via link speed (The faster the link the lower the metric
If the metrics are the same it will use the first detected NIC - configured through adapter settings

What this means is that if you have two similar speed interfaces the primary method of configuring priority is through the following interface:

In the above example if the metrics are both the same the interface named "Ethernet" will take priority.

Refereces:
http://technet.microsoft.com/en-us/library/cc779696%28v=ws.10%29.aspx
http://w3facility.org/question/routing-decision-when-there-are-two-default-gateways-on-windows/
http://superuser.com/questions/198544/how-does-windows-7-decide-which-route-to-take-if-2-connections-to-an-internet-so
http://serverfault.com/questions/59442/routing-decision-when-there-are-two-default-gateways-on-windows
http://theregime.wordpress.com/2008/03/04/how-to-setview-the-nic-bind-order-in-windows/
http://techrena.net/view-change-network-adapter-card-priority-binding-order-windows-7/
http://windows.microsoft.com/en-ca/windows/change-network-protocol-bindings-order#1TC=windows-7

Network Ubuntu Install via PXE Using Preseed - Kickstart Alternative

2014-11-11T09:50:00.000-05:00

Ubuntu has a method of automatic network installation similar to redhat's kickstart install.

It's called "debian-installer" and it is the recommended method of installing ubuntu automatically.

Similar to kickstart you have a preconfiguration file called "preseed" in which you answer questions asked by the installer.

You can either:

Install ubuntu manually and export answers to questions
Create answers to questions manually

#1 you can perform using:



# Generate a sample preseed.cfg with some default values you selected during manual install

apt-get install debconf-utils

debconf-get-selections --installer > preseed.cfg

debconf-get-selections >> preseed.cfg

Then you make this file available in the kickstart menu.

I've included a sample menu file (which points to the preseed) as well as the contents of a basic preseed file.
Menu File:



default linux



SERIAL 0 115200



label linux

kernel images/ubuntu/14.04/install/netboot/ubuntu-installer/amd64/linux

append initrd=images/ubuntu/14.04/install/netboot/ubuntu-installer/amd64/initrd.gz auto=true priority=critical interface=auto url=ftp://fbsd-ftp/images/ubuntu/14.04/preseed/sandvine.seed -- console=ttyS0,115200n8

IPAPPEND 2

Preseed File:



# Primary network interface:

d-i netcfg/choose_interface select auto



# Locale Configuration

d-i debian-installer/locale string en_US



# Set Keyboard to American English

d-i console-keymaps-at/keymap select American English

d-i debian-installer/keymap string us



# Set timezone

d-i time/zone string American/Toronto



# Do not make a user account

d-i passwd/make-user boolean false



# Assign the password for root

d-i passwd/root-password password sandvine

d-i passwd/root-password-again password sandvine



# Allow root login

user-setup-udeb passwd/root-login boolean true



# Partitioning

# Partition using LVM on the first and only detected disk

d-i partman-auto/method string lvm

# If one of the disks that are going to be automatically partitioned

# contains an old LVM configuration, the user will normally receive a

# warning. This can be preseeded away...

d-i partman-lvm/confirm boolean true

d-i partman-lvm/confirm_nooverwrite boolean false

partman-lvm partman-lvm/device_remove_lvm boolean true

partman-lvm partman-lvm/confirm_nochanges boolean true

partman-lvm partman-lvm/confirm boolean true

unknown partman-lvm/confirm boolean true

partman-lvm partman-lvm/vgdelete_confirm boolean true



# Choose automatic partitioning recipes where automatic puts

# All configuration on the same partition

d-i partman-auto/choose_recipe select automic

# Complete partitioning without further confirmation

d-i partman-partitioning/confirm_write_new_label boolean true

d-i partman/choose_partition select finish

d-i partman/confirm boolean true

d-i partman/confirm_nooverwrite boolean true



#Choose some additional packages to instal

d-i pkgsel/include string openssh-server





### Finishing up the installation

# During installations from serial console, the regular virtual consoles

# (VT1-VT6) are normally disabled in /etc/inittab. Uncomment the next

# line to prevent this.

d-i finish-install/keep-consoles boolean true



# Send command to pxeboot server to tell it to boot from HDD after imaging is successful

d-i preseed/late_command string /bin/echo "$(hostname -s)" | nc -w 3 lab-nfsbootlinux 2509



# Avoid that last message about the install being complete.

d-i finish-install/reboot_in_progress note

Ubuntu does have support for kickstart but there are some important features that may not be fully featured in kickstart that are available in debian installer. Ubuntu even has system-config-kickstart utility to build kickstart files but because of the lack of full functionality it does not end up being as useful as the more fully funcitoning debian installer.

References:
Serial Console - http://blather.michaelwlucas.com/archives/638
Package Install (openssh) - http://www.stratuslab.eu/fp7/doku.php/tutoral:examplepreseed.html
Remove LVM Parts - http://ubuntuforums.org/showthread.php?t=1505174
Debian Example Preseed - http://d-i.debian.org/manual/example-preseed.txt
Ubutntu Example Preseed - https://help.ubuntu.com/lts/installation-guide/example-preseed.txt
Ubuntu 12.04 Preseed Explained - https://help.ubuntu.com/12.04/installation-guide/i386/preseed-contents.html
Ubuntu 10.04 Preseed example - https://help.ubuntu.com/10.04/installation-guide/example-preseed.txt
Generate a preseed - http://searchitchannel.techtarget.com/feature/Performing-an-automated-Ubuntu-install-using-preseeding
Puppet and Preseed - http://ghantoos.org/2012/10/21/cocktail-of-pxe-debian-preseed-ipmi-puppet/
Add Console and debug to menu - http://christian.hofstaedtler.name/blog/2008/09/debian-installer-preseeding-autostart-from-pxe.html
Ubuntu 14.04 install bug - http://www.michaelm.info/blog/?p=1378
What you need to append to menu - https://saz.sh/2011/07/30/preseed-debian-squeeze-using-pxe/
Create User Accounts - https://help.ubuntu.com/community/Cobbler/Preseed
Debian links:
https://www.debian-administration.org/article/394/Automating_new_Debian_installations_with_preseeding
https://www.debian.org/releases/wheezy/i386/apbs04.html.en
https://www.debian.org/releases/wheezy/i386/apbs03.html.en

Official Documentation:
http://www.debian.org/releases/stable/i386/apb.html.en
https://wiki.debian.org/DebianInstaller/Preseed
http://searchitchannel.techtarget.com/feature/Performing-an-automated-Ubuntu-install-using-preseeding
http://www.debuntu.org/how-to-unattended-ubuntu-deployment-over-network-page-5-preseed-file/

New Docs:
http://web.theurbanpenguin.com/auto-installing-ubuntu-16-04/
https://www.debian.org/releases/stable/example-preseed.txt
http://hands.com/d-i/

How to get a post-install command working:
http://www.50ply.com/blog/2012/07/16/automating-debian-installs-with-preseed-and-puppet/
https://ubuntuforums.org/showthread.php?t=1977570

Splitting and Combining Files

2014-08-25T09:24:00.001-04:00

It's a fairly common scenario that you may have files larger than the existing storage medium you have. So you can buy multiple storage items, but you'll need to split the file out into smaller chunks to be able to store it in multiple locations. How do you do this? In linux it's fairly easy using the split utility. You can even recombine and split in windows fairly easily.

The initial split is done using the split command as follows:

split -b 1024m file file.part-

Where -b is the block size you want to split by. In this example 1GB.
file is the original file you want to split
file_ is the prefix you would like to use.

When running the split command the files it will create will be of the format

file.part-aa
file.part-ab
file.part-ac
etc.

As defined by the prefix above.

To recombine these files you can use cat in linux to concatenate the existing files.

cat file-part-* > file

In windows you can use the copy command

copy /b file.part-aa + file.part-ab + file.part-ac + file.part-ad file

I'm unsure of how to specify regex for the copy command in windows but this is a quick and dirty way to get it done and you could always generate a list of files and manually script it together.

You can also pipe the output of gzip into split to compress the archive but you will probably have additional overhead of decompression, and if you don't care so much about space savings split is probably going to be quicker.

An alternate way of doing this would be to use the dd utility:

Example file

tmp

contents:

cat tmp
a
b
c
d

Sizing details (in bytes):

ls -ls tmp
0 -rw-r--r--  1 bnold  domusers  8 Aug 25 15:20 tmp

wc -c tmp
       8 tmp

Split

dd if=tmp of=tmp.part1 bs=1 count=4 
dd if=tmp of=tmp.part2 bs=1 count=4 skip=4

Restore

dd if=tmp.part1 of=tmp_new bs=1 count=4
dd if=tmp.part2 of=tmp_new bs=1 count=4 seek=4

Contents of restored file:

cat tmp_new
a
b
c
d

Size of restored file:

ls -ls tmp_new
0 -rw-r--r--  1 bnold  domusers  8 Aug 25 15:44 tmp_new

wc -c tmp
       8 tmp

Validating integrity:

md5 tmp
MD5 (tmp) = 47ece2e49e5c0333677fc34e044d8257
md5 tmp_new
MD5 (tmp_new) = 47ece2e49e5c0333677fc34e044d8257

Hashes match, we're good.
References:
http://www.linuxquestions.org/linux/answers/applications_gui_multimedia/splitting_and_merging_files_using_dd
http://en.wikipedia.org/wiki/Split_%28Unix%29
http://stackoverflow.com/questions/1120095/split-files-using-tar-gz-zip-or-bzip2
http://linuxpoison.blogspot.ca/2008/09/split-and-merge-large-files.html
http://serverfault.com/questions/86808/break-up-a-dd-image-into-multiple-files

Boot Directly into New Kernel After Kickstart Install

2014-08-14T16:19:00.000-04:00

It is useful to be able to boot directly into your freshly installed linux box immediately after installation.

However there are multiple different methods to going about doing this, both depending on which version of GRUB you are running, because the information you will require to make the next kernel load will depend on what you plan on booting next.

Both instructions will require the use of post-install scripts to pull data from the filesystem about next boot parameters.

First off, it's a good idea to get some debug going on your console screen, which can be done via the following:

%post
exec < /dev/console > /dev/console
echo "# Running Post Configuration #"
#....Do stuff....
echo "End of post-install steps"

References: http://blog.mozilla.org/jv/2011/01/31/redirecting-kickstart-prepost-over-serial-console/

GRUB 0.97

In GRUB 0.97 its fairly easy, you just grab the cmdline options (arguments) from the /boot/grub/grub.conf

cmdline=$(awk /kernel.*console/'{$1=$2=""; print$0}' /boot/grub/grub.conf)                       
kexec -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname -r).img --append="${cmdline}"
sleep 2                                                                                          
kexec -e

GRUB 2

In GRUB 2 its a bit more challenging as each GRUB menu is dynamically generated from /etc/grub2.cfg which points to /etc/grub.d/* files which have scripts that parse the underlying filesystem to locate kernels to dynamically build the grub menu

ls -lah /etc/grub.d/
total 80K
drwx------.  2 root root 4.0K Aug 12 13:16 .
drwxr-xr-x. 99 root root 8.0K Aug 12 20:09 ..
-rwxr-xr-x.  1 root root 8.5K Jun 30 12:16 00_header
-rwxr-xr-x.  1 root root 9.3K Jun 30 12:16 10_linux
-rwxr-xr-x.  1 root root  11K Jun 30 12:16 20_linux_xen
-rwxr-xr-x.  1 root root 2.5K Jun 30 12:16 20_ppc_terminfo
-rwxr-xr-x.  1 root root  11K Jun 30 12:16 30_os-prober
-rwxr-xr-x.  1 root root  214 Jun 30 12:16 40_custom
-rwxr-xr-x.  1 root root  216 Jun 30 12:16 41_custom
-rw-r--r--.  1 root root  483 Jun 30 12:16 README

By default linux grub menus are generated via /etc/grub.d/10_linux and take the defaults for cmdline options from /etc/defaults/grub

Grabbing the cmdline

cmdline=$(awk -F '"' '/GRUB_CMDLINE_LINUX/{print$2}' /etc/default/grub)

Grabbing the root Volume

I haven't figured out a dynamic way of grabbing the root volume yet so I just manually stuff in /dev/mapper/(volume-group-lv) in

Putting GRUB2 all together to kexec the kernel

%post                                    
exec < /dev/console > /dev/console       
echo "# Running Post Configuration #"                                                                                    
cmdline=$(awk -F '"' '/GRUB_CMDLINE_LINUX/{print$2}' /etc/default/grub)                                               
kexec -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname -r).img --append="root=/dev/mapper/VolGrp-Vol1 ro {cmdline}"                                                                                                               
sleep 2                                                                                                               
echo "Loading new Kernel"                                                                                             
kexec -e 
%end

Synology USB Drive Mapping

2014-04-12T19:13:00.000-04:00

Synology will automatically name usb drives the order in which you plug them in.

The first drive you plug in will be mapped to usbshare1, the second usbshare2 and so on.

I've found that after updates the ordering can get confused where usbshare1 will become mapped to usbshare3. This is due to some confusion in the usb mapping file.

In order to rebuild the order you will have to follow these steps:

Enable ssh or telnet accessibility and remote in

Move the mapping file

mv /usr/syno/etc/usbno_guid.map /usr/syno/etc/usbno_guid.map.old

Reboot
Plug in the drive you want to be mapped usbshare1 first, then follow the order until you hit the desired ordering

Change TTYs In Linux To Debug

2013-12-10T09:42:00.002-05:00

I have found this useful while doing a kickstart install. Being able to change your terminal when an irrecoverable error shows up is very useful.

The way to get to other terminal types is to use a combination of: Ctrl+Alt+F-key

Here's the functions of some of them:

Ctrl+Alt-F1 - The installation dialog when using text or cmdline 
Ctrl+Alt-F2 - A shell prompt
Ctrl+Alt-F3 - The install log displaying messages from install program
Ctrl+Alt-F4 - The system log displaying messages from kernel, etc.
Ctrl+Alt-F5 - All other messages 
Ctrl+Alt-F7 - The installation dialog when using the graphical installer

Network Serial Ports In VMware (Licensing Issues)

2013-11-25T13:30:00.000-05:00

As per 5.1 Documentation it says you should be able to connect to a guest serial port by attaching a virtual serial port and using the "Use Network" option as of 5.x.

What they do not mention is that this feature only works in VMware Enterprise licenses.

I spent hours trying to figure out why my serial port would never maintain its "Connected" checkbox as shown below:

It would consistently revert and I had already confirmed that firewall was not blocking the ports:

~ # esxcli network firewall ruleset list -r remoteSerialPort
Name              Enabled
----------------  -------
remoteSerialPort     true

I have also verified that the esxi server is accepting connections to 3008 on the host:

telnet RPC-D8-03VM 3008   
Trying 10.129.36.2...                                
Connected to RPC-D8-03VM.phaedrus.sandvine.com.      
Escape character is '^]'.                            
                                                     
                                                     
test                                                 
                                                     
                                                     
testing                                              


~ # nc -l 3008


test


testing

But nothing seemed to work. Even new version of virtual hardware/different OS.

lspci wasn't showing anything even being connected to the guest.

After contacting VMware support they looked through some logs and noticed the following line stating this feature wasn't licensed:

2013-11-25T17:53:07.636Z| vmx| I120: Msg_Post: Error
2013-11-25T17:53:07.636Z| vmx| I120: [msg.serial.uri.nolicense.basic] The currently available license does not permit the use of remote serial ports.

Due to this licensing issue the VM will not even attempt to add the virtual serial port and you will not be able to telnet to your serial port or use a virtual serial port concentrator without an Enterprise license.

For future reference the log that told us that this was not a licensed feature was:

/vmfs/volumes/path/to/vmx/vmware.log

The fact that the GUI does not give any kind of notification of this functionality not working is a bit disappointing, but not more so than the functionality being licensed and not being usable.

References to others having this issue: http://community.spiceworks.com/topic/247096-virtual-serial-ports-in-vsphere-esxi-5

ESXi Network Port Statistics

2013-10-04T15:46:00.002-04:00

To get port statistics you first need to know which port you wish to investigate via its port-id.

This can be gathered using the following method:

esxcli network vm list
World ID  Name            Num Ports  Networks
--------  --------------  ---------  ----------------
   54526  TPC-F11-09-013          2  VM Network, 2/13
   54675  TPC-F11-09-019          2  VM Network, 2/19
   55055  TPC-F11-09-016          2  VM Network, 2/16
   59076  TPC-F11-09-010          2  VM Network, 2/10
   69868  TPC-F11-09-011          2  VM Network, 2/11
   71906  TPC-F11-09-004          2  VM Network, 2/4

Grab the “World ID”

Use the world ID to get the port-id:

esxcli network vm port list -w 54526
   Port ID: 50331665
   vSwitch: vSwitch1
   Portgroup: 2/13
   DVPort ID:
   MAC Address: 00:50:56:82:0b:97
   IP Address: 0.0.0.0
   Team Uplink: vmnic5
   Uplink Port ID: 50331650
   Active Filters:

From here we can correlate which port-id is giving us the most traffic or which ones is giving us problems. Once we have the port-id we can get some statistics on that port.

esxcli network port stats get -p 50331665
Packet statistics for port 50331665
   Packets received: 554581
   Packets sent: 473571
   Bytes received: 740098057
   Bytes sent: 91336282
   Broadcast packets received: 9909
   Broadcast packets sent: 15
   Multicast packets received: 0
   Multicast packets sent: 0
   Unicast packets received: 544672
   Unicast packets sent: 473556
   Receive packets dropped: 3
   Transmit packets dropped: 0

References: http://blogs.vmware.com/vsphere/2013/01/network-troubleshooting-using-esxcli-5-1.html

Network Performance

2013-07-19T13:19:00.000-04:00

dd with netcat

Local (Sending Server)

dd bs=16M if=/dev/zero | nc $ip_address $port#

Remote (Listening Server)

nc -l $port# | dd bs=16M of=/dev/null

reference material + compression: http://www.ndchost.com/wiki/server-administration/netcat-over-ssh

Testing MTU

2013-07-10T10:42:00.002-04:00

MTU (Maximum Transmission Unit) is the maximum effective size of a PDU (Protocol Data Unit) that can be transmitted onwards by a layer.

These configuration maximums may need to differ to accommodate different traffic types. They may need to be increased or decreased depending on whether you have configured VPN and require additional header area to transmit data as your WAN link may be configured for 1500 MTU and your VPN may require an extra 8 bytes of data to package your VPN frames. If this is the case you may need to decrease your MTU to 1492.

An easy way to test to test and validate your MTU sizes is by using a simple ping with the DF (do not fragment bit).

Windows:
The following will send a frame with a size of 1500 bytes (using -l) and will tell windows not to fragment the packet (-f do not fragment)

C:\Users\user>ping $hostname -l 1500 -f

Pinging $hostname [$ip_address] with 1500 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for $ip_address:
    Packets: Sent = 3, Received = 0, Lost = 3 (100% loss),
Control-C
^C

Linux:
This will do the same as the windows command (-M do will tell Linux to not fragment) it only does one attempt to prevent spam (-c 1) and will send with a size of 1500 bytes (-s 1500)

# ping -s 1500 -M do $hostname -c 1
PING $hostname ($ip_address) 1500(1528) bytes of data.
From TPC-F3-05.phaedrus.sandvine.com ($ip_address) icmp_seq=1 Frag needed and DF set (mtu = 1500

--- $hostname ping statistics ---
0 packets transmitted, 0 received, +1 errors

FreeBSD:
FreeBSD ping with no fragment (-D) and a size of 1500 bytes (-s 1500)

sudo ping -D -s 1500 $hostname
PING $hosntmae ($ip_address): 1500 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long

Here's some commonly known and referenced MTU sizes

1500 The biggest-sized IP packet that can normally traverse the Internet without getting fragmented. Typical MTU for non-PPPoE, non-VPN connections. 
1492 The maximum MTU recommended for Internet PPPoE implementations. 
1472 The maximum ping data payload before fragmentation errors are received on non-PPPoE, non-VPN connections. 
1460 TCP Data size (MSS) when MTU is 1500 and not using PPPoE. 
 
1464 The maximum ping data payload before fragmentation errors are received when using a PPPoE-connected machine. 
1452 TCP Data size (MSS) when MTU is 1492 and using PPPoE. 
576 Typically recommended as the MTU for dial-up type applications, leaving 536 bytes of TCP data. 
48 The sum of IP, TCP and PPPoE headers. 
40 The sum of IP and TCP headers. 
28 The sum of IP and ICMP headers.

http://en.wikipedia.org/wiki/Maximum_transmission_unit
http://www.dslreports.com/faq/695

Tunneling vSphere Client in Windows

2012-02-14T11:36:00.000-05:00

Using tunneling over SSH you can forward the multiple ports needed to access the vSphere client used to administrate ESX or ESXi hosts.

Requirements:

SSH jumpbox (from this jumpbox you should have access to the ESX/ESXi machine)
PuTTY or Cygwin

So basically we are binding the following ports: 443, 902, and 903 to our localhost using PuTTy in this tutorial.

Steps:

Open up PuTTy and configure the settings to look as follows where destIP=your destination IP address of the ESX box

ssh -L 443:destIP:443 -L 902:destIP:902 -L 903:destIP:903 user@jumpboxIP

Edit the hosts file in windows C:\Windows\system32\drivers\etc\hosts (this is required as esxi relies on dns name and uses local resolution).

Add the entry:
127.0.0.1 ESXiHostname
Open up vSphere client and login with the hostname of the remote ESX box

Common Errors:
Unable to connect to the MKS: Failed to connect to server :902

Occurs if you are not properly forwarding port 902 903