For Performance Engineers

Access S3 files meta data and other details using Python Boto using Assume Role

2014-06-04T03:15:00.001-07:00

Here is the code

import sys
import boto
from boto.sts import STSConnection
from boto.s3.connection import S3Connection
from boto.s3.key import Key

sts_connection = STSConnection()
assumedRoleObject = sts_connection.assume_role(
role_arn="arn:aws:iam::0000000000:role/xxxx/xxxx-yyyy-zzz",
role_session_name="AssumeRoleSession1"
)

s3_connection = S3Connection(
aws_access_key_id=assumedRoleObject.credentials.access_key,
aws_secret_access_key=assumedRoleObject.credentials.secret_key,
security_token=assumedRoleObject.credentials.session_token
)

bname="BUCKETNAME"
fname="FILENAME"
bucket = s3_connection.get_bucket(bname)
files=bucket.list(prefix=fname)

for file in files:
k=bucket.get_key(file)
print k.get_metadata('key-name')
print file.name
print file.size
print file.last_modified
print file.get_acl()
print

1: Lets assume your key name is x-amz-meta-keyname then in the code pass your keyname as "keyname". No need to mention x-amz-meta

Let me know in comments section if his works/doesn't work for you

"Unable to execute HTTP request: Connection to https://ec2.amazonaws.com refused" error while executing AWS Java SDK Sample projects

2013-09-18T06:43:00.003-07:00

If you are behind proxy then these sample projects will not read proxy from eclipse proxy settings.

to fix this issue you have to add some extra lines of code into your sample projects.

Create a object of ClientConfiguration class and set proxyhost and proxyport properties and then pass this object while creating client (ec2,S3 or sdb etc)

so your init method should look something like this after modification

private static void init() throws Exception {
/*
* This credentials provider implementation loads your AWS credentials
* from a properties file at the root of your classpath.
*
*/

ClientConfiguration clientCfg = new ClientConfiguration();
clientCfg.setProtocol(Protocol.HTTP);

//setup proxy connection:
clientCfg.setProxyHost("HTTP_PROXY_IP");
clientCfg.setProxyPort(8080);

AWSCredentialsProvider credentialsProvider = new ClasspathPropertiesFileCredentialsProvider();

ec2 = new AmazonEC2Client(credentialsProvider,clientCfg);
s3 = new AmazonS3Client(credentialsProvider,clientCfg);
sdb = new AmazonSimpleDBClient(credentialsProvider,clientCfg);
}

"Missing requirement: Eclipse Data Tools Platform Amazon SimpleDB UI Plug-in 1.0.0.v201308121803" Error while installing AWS Toolkit for Eclipse

2013-09-18T05:55:00.000-07:00

In Eclipse follow below steps to fix this

1: Help -> install new software
2: Type "http://download.eclipse.org/releases/kepler" in the url field
3: Select "Database Development" and install from the list.
4: Try to re install AWS Toolkit, it will work.. :)

Get count of messages in a jboss message queue from command prompt

2011-03-30T23:11:00.000-07:00

Hi All,

You can also retrieve message count in a particular message queue of jboss, from command prompt, without accessing jmx console.

With command prompt you can automate and send alerts in case message count is above your application threshold

The command to do this is

./twiddle.sh -s localhost:1099 get jboss.messaging.destination:service=Queue,name=ResponseQueue MessageCount

Run this command from jboss bin directory and you can change queue name (in above example it is ResponseQueue) to your application specific queue name

Run Multiple Jboss on Same server

2011-02-25T04:15:00.000-08:00

Hi All,

Its very easy to run multiple jboss on same server.

You just need to add one parameter wen you are starting your jboss.

the paramter is

-Djboss.service.binding.set=ports-01

example

./run.sh -c myproj -Djboss.service.binding.set=ports-01 &

to start another jboss on same server use

./run.sh -c myproject -Djboss.service.binding.set=ports-02 &

Just change -Djboss.service.binding.set parameter and you can start many jboss on same server, without any changes in any conf file.

Install perl GD module

2010-12-14T04:13:00.001-08:00

Hi all,

If you are getting below error while instaling perl GD module module

Could not find gdlib-config in the search path. Please install libgd 2.0.28 or higher

Follow the below steps

1:do yum install gd gd-devel
2: go to cpan
3:then fire install GD

You are all set to go, your GD module should install now

How to Log slow queries in postgresql database

2010-07-07T02:41:00.000-07:00

Hi All,

Postgresql db can log sql queries that takes more time to execute then specified time limit.

Suppose you want to log sql queries that takes more than 100ms to execute.

Follow the below steps

1: Open postgresql.conf file. You will find this file in data folder of postgres.
2: Modify the below paratmeters

a:logging_collector = on
b:log_min_duration_statement = 100
c:log_duration = on

3: Save and close this file.
4: Restart the DB

Now execute the queries again. P{ostgres will log all the queries in pg_log folder
pg_log folder will be inside your data folder of postgresql.
Name of log file will be in the following format

postgresql-%Y-%m-%d_%H%M%S.log

If you want to change log directory and log file name, just modify these parameters to desired value

#log_directory = 'pg_log' # directory where log files are written,
# can be absolute or relative to PGDATA
#log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,

Get Index statistics in Postgresql database

2010-07-07T00:02:00.000-07:00

I always observed that around 30 to 40% indexes created on a database are never used by any queries.

Having extra indexes on a DB, can degrade the performance (update and insert statements) , consumes lot of space and can increase the time of backup and restoring the DB.

It make sense to check the index utilization.

To check the same in Postgresql Database follow below steps.

1: Open postgresql.conf file. This file is inside your data directory of postgres.
2: Change following parameters

A: logging_collector = on
B:track_activities = on
C:track_counts = on
3: save and close the file
4: Restart the postgresql server

Now do all possible operartions using application, just to make sure all possible queries got fired to database.

Now just run the following query

select * from pg_stat_all_indexes where schemaname= order by relname,idx_scan desc;

You will find statistics of all indexes.

Always make sure your JVM is runnig is SERVER mode not in client mode

2010-04-11T22:09:00.000-07:00

Hi,

JVM can run in many modes and selecting a proper mode can help improve your application performance

In servers, make sure JVM is runnig in server mode . This activates the server hotspot JIT.
It may need more time to start. But this delivers best performance.

Other modes of JVM are client and classic mode.

make sure -server, should be the first argument to java

Too Many Open Files problem in Linux

2010-03-28T23:12:00.000-07:00

Hi All,

Most of the time during stress testing on linux server, you may encounter the following problem

SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=80] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files

We get this error when system reaches the the max file limit set on your Linux OS

first check the limit set on your OS with the following command
cat /proc/sys/fs/file-max

if you want to check the current open files .
1:lsof |wc -l   ( to check number of all open files)
2:lsof -p  | wc -l  ( to check number of opened files by a process)

in Ubuntu
ulimit -n



to change the value
1 edit /etc/sysctl.conf and set fs.file-max to any desired value

then run

sysctl -p

you may have to restart your system after this

troubleshoot mysql database connection pool problems

2010-03-22T01:20:00.000-07:00

If you are getting following error messages, then its time to tune your mysql max_connection value

1:User 'mysqldba' has exceeded the 'max_user_connections' resource
2:Too many connections
3:GenericJDBCException: Cannot open connection
4:message from server: "Too many connections"

Now once u got these messages, we need to change value of max_connection variable in mysql database.

There are two ways to do this.

Method 1:Without restarting the mysql DB.
a;login to the mysql db.
b: run the query ----
show variables like "max_connections";
c: this will display the current value to max_connections.
d: Now to change the value of max_connection, run this query
set GLOBAL max_connections=NewValue;

max_connection value is changed , now again run the query
'show variables like "max_connections";'
to check if the value is reflecting.

But this value will be temporary, if you restart the mysql DB, max_connection value will again set to its default value

Method 2: This requires restarting the DB, but this change will be permanent and will not go even after restart of the db

a: For this you need to change the global setting file of mysql ie "my.cnf" file
b: By default, path of my.cnf will be under etc directory. If you dont know the path, run the
following command to get the path of my.cnf file
find / -name my.cnf
c: Once you got the path . open the file in vi editor.
d: You will find [mysqld] inside this file
e: Just add one line below this

max_connections=NewValue
f: restart the db.
g: login to the mysql and run the following query to check the value of max_connection

show variables like "max_connections";

Monitor cpu utilization of all threads in a process in windows

2010-01-13T21:32:00.000-08:00

Hi All,

We can monitor cpu utilization of all threads for a process in windows using perfmon tool and this tool also writes logs to a file.
We can also use process explorer for this but this tool can not write logs.

To monitor threads in a process in windows,
1: Launch perfmon
2:Click Performance Logs and Alerts tree on left most window
3:Select Counter Logs from tree structure
4: Right click Counter Logs and select "New Log setting"
5: Give any name for new logs.
6: Click on Add general tab at top.
7: click "add counters" button.
8:From "Performane objects" drop down, select thread.
9:At right hand side, it will display all currently executing threads in windows.
10:Select the threads, you want to monitor and counter to monitor from lft hand side window.
11:Click close and go to "Log Files" tab.
12:Set the log file path and type.

Tomcat Performance parameters

2009-09-22T02:21:00.000-07:00

1:DNS Lookups

When your webserver logs the information about the client, it can either log
the client’s numeric IP address or look up the actual host name in the Domain Name
Service. DNS lookups require network traffic, involving a round-trip response
from multiple servers, possibly far away and possibly inoperative, resulting in delays.

find this part of the server.xml file:

Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="true" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />

Just change the enableLookups value from "true" to "false", and restart Tomcat. No
more DNS lookups and their resulting delays!

2: Number of threads

We can control the number of threads that are allocated by changing a Connector’s
minThreads and maxThreads values.This setting is totally depend upon traffic that your application handles.

3: Precompile your JSP

When a JSP is first accessed, it is converted into Java servlet source code, which must then be compiled into Java bytecode.

For example, if you want index.jsp (in the root of your webapp’s directory) to always
be precompiled at webapp startup time, you can add a tag for this file in
your web.xml file, like this:

?servlet?
?servlet-name?index.jsp?/servlet-name?
?jsp-file?/index.jsp?/jsp-file?
?load-on-startup?0?/load-on-startup?
?/servlet?

Enjoy tomcat!!!!!!!!!

run a command in solaris from winodw and output will save in window through batch file

2009-08-18T22:21:00.000-07:00

Hi All,

some time we need to run some command in solaris or Unix machine and need to collect output in window.
This all can be automated and directly we can invoke a shell scrpit in solaris from windows. You need putty to do this. Your shell script should be in windows machine.

CODE:

Create a batch file (suppose putty is saved in d: drive of windows and your shell script that you want to run is saved in d: drive).shell script name is "shellscript.sh"

D:\putty -ssh username@servername -pw mypassword -m "D:\shellscript.sh"
ftp -i -s:"D:\ftp.txt"

now create a ftp.txt file in D: drive as follows

OPEN servername
username
password
get filename
BYE

Thanks
Shashank

Write a parameter to a text file in loadrunner script

2009-07-09T01:57:00.000-07:00

char *filename = "c:\\myfilename.txt";
long file_stream;
vuser_init()
{

if ((file_stream = fopen(filename, "a+")) == NULL) //open file in append mode
{
lr_error_message ("Cannot open %s", filename);
return -1;
}

fprintf (file_stream, "%s\n", lr_eval_string("THE PARAMETER IS :{Parameter}")); //Parameter is parameter name to be written in a file

Return 0;

}

vuser_end()
{
fclose(file_stream);//dont forget to close file_stream in vuser_end scetion
return 0;
}

monitor cpu and memory utilization of an individual process in windows

2009-07-02T06:08:00.000-07:00

Sometime it is required to measure cpu utilization of a process instead to cpu utilization of whole machine in windows.
This is very much possible with PerfMon tool in windows
Steps to follow
1: Launch perfmon
2:Click Performance Logs and Alerts tree on left most window
3:Select Counter Logs from tree structure
4: Right click Counter Logs and select "New Log setting"
5: Give any name for new logs.
6: Click on Add objects tab.
7:Select Process from the list then click on Add then close.
8:Click "Log Files" tab at the top.
9:Select Log File Type as csv or tsv,as per your requirement
10: You are all set to go by default log file will be saved in c:\perfLogs folder

Code for puttting custome Pacing in script

2009-06-09T23:14:00.000-07:00

Action()
{
merc_timer_handle_t timer;
double duration;
double pacing_time;

// Read desired pacing timer from a text file.
pacing_time = (double)jds_read_pacing_time("C:\\TEMP\\vugen_pacing.txt");

// Start the timer.
timer = lr_start_timer();

//Your Action Code

// Stop the timer
duration = lr_end_timer(timer);

// Wait for the necessary number of seconds to elapse
if (duration < pacing_time) {
lr_think_time(pacing_time - duration);
} else {
lr_error_message("Pacing time exceeded. Target: %G seconds. Actual: %g seconds", pacing_time, duration);
}

return 0;
}

// Read pacing time from a file. Returns time in seconds (whole seconds only).
// Note that file can be on a shared network drive.
int jds_read_pacing_time(char* file_name) {
long fs; // file stream
int number_from_file;

// Open the file for reading
fs = fopen(file_name, "r+");
if (fs == NULL) {
lr_error_message("Error opening file: %s", file_name);
return -1;
}

// Read number from file.
if (fscanf(fs, "%d", &number_from_file) != 1) {
lr_error_message("Error reading number from file: %s", file_name);
return -1;
}

fclose(fs);

return number_from_file;
}

URL vs HTML recording mode

2009-06-09T01:59:00.000-07:00

HTML based mode, records script for every user action that is
performed during recording, while URL based mode records each and
every browser request to the server and resources received from the
server.

IN HTML recording mode the execution will be fast..as all
the server responses will not be captured in the script.
In URL all the server responses will be captured in the
script and hence the execution will be slow

how to monitor tomcat 6.0.16 with jconsole

2009-06-09T01:58:00.001-07:00

go to tomcat bin folder
click on tomcat6w.exe
go to java tab
go to java options
type :
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8090
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
you are all set to go
now you can monitor your tomcat through jconsole and give port as 8090

back or referesh error with siebel protocol in loadrunner

2009-06-09T01:57:00.001-07:00

hi this is a very common problem everyone facing while working on
siebel protocol of loadrunner.
this problem is coming from 2nd iteration .

solution is very simple

run the script and see in which web_url or web_submit, this problem is
coming soo just change mode to http from html and resources from 0
to 1.

I hope this will surely help you

How to convert Web based http Script to Java Vuser Script

2009-06-09T01:49:00.001-07:00

Hi All,

loadrunner has two file

1: sed.exe (in loadrunner bin directory)
2: web_to_java.sed (in loadrunner dat directory)

with the help of above files we can convert a WEB(http/html) script
to java script

STEPS:

1:Record your script using WEB(HTML/HTTP) protocol.
2:Add Loadrunner Bin directory to your PATH enviroment variable.
3:Copy a part of your script to a text file (for example script.txt).
4:Go to loadrunner DAT directory.
5: run the following command

sed -f web_to_java.sed "path of your script txt file" > "Destination
java file path"

ex:
sed -f web_to_java.sed script.txt >java.txt

Points to remember

1: Java scrpit use "<>" parameter braces and httpl/html uses "{ }".
Above command will not replace this. so manually we need to change
this. Above command will put following warning before each line that
uses parametrization

"Suspected parameterization in next line. If true make sure to
substitute {} with proper parameter braces on the parameter".