Learn it

Large File Support in Linux for C/C++ operations

2008-02-19T15:06:00.000-08:00

Overview

My current project deals with high definition movies that are encrypted and used for IPTV. The problem occurred when I tried to encrypt a real world 4.8GB movie. The application server (OC4J) crashed with the error:
File size limit exceeded$JAVA_HOME/bin/java $JVMARGS -jar $OC4J_JAR $CMDARGS

Limitations

In a C/C++ application the file operations can be handled using the fcntl.h header file. It provides operations for opening, writing to a file and many more. The size of every file is stored in a variable of type off_t. For 32-bit systems the maximum value for off_t is 2³¹thus limiting the maximal file size to 2³¹ bytes (2 GiB). For 64 bit systems like x86-64 this maximum value is much greater and they have support for large files with size up to 2⁶³ bytes.

Prerequisites

The LFS support is done by the Linux kernel and the GNU C library (glibc) and is implemented since version 2.4.0 of the Linux kernel and glibc 2.2.3 (e.g. SuSE 7.2, Red Hat 7.1). The file system is also important - ext2/ext3 have full support for LFS.

OS Configuration

The current configuration of the OS should also be checked. All resource and process limitations can be examined and changed using the Linux command ulimit e.g.:

$ulimit -a

...

file size (blocks, -f) unlimited

The "file size" property shows the maximum size of a file in bytes that you can manipulate - probably a very large number or unlimited. If it's not use the same command to change it to any number (e.g. 5000000 bytes) or unlimited:
$ulimit -S -f unlimited
After that you can create a large file for a test. This creates around 5GB file:
$dd if=/dev/zero of=outputfile bs=1M count=5

Resolutions

Compile your programs with "gcc -D_FILE_OFFSET_BITS=64" or "g++ -D_FILE_OFFSET_BITS=64" for C++ code. This forces all file access calls to use the 64 bit variants. It's important to always use the correct data types and to not use e.g. int that is 32 bit instead of off_t (file size). For portability with other platforms you should use getconf LFS_CFLAGS which will return -D_FILE_OFFSET_BITS=64 on Linux platforms but might return something else on e.g. Solaris. For linking, you should use the link flags that are reported via getconf LFS_LDFLAGS. On Linux systems, you do not need special link flags.
Define _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE. With these defines you can use the LFS functions like open64 directly.
Use the O_LARGEFILE flag with open to operate on large files.

I chose the first solution and changed the build script that compiles all components (to links them into a shared binary library later) :

g++  -D_FILE_OFFSET_BITS=64 ...

No flag is needed during the linkage phase. This approach do not require code changes!

Implemetation

A sample C/C++ implementation follows:
- Import required headers:
#include "fcntl.h"
- Open file with name outputFileName for create/write:

int hOutputMovie = open(outputFileName, O_CREAT | O_WRONLY, S_IRWXU);
if (-1 == hOutputMovie)
{
 loggerVpr.log("Problem opening file %s", outputFileNameChars);
 ...
}

- Write to the file some buffer of data (unsigned char *buffer) with specified length (int len):

if (len != write(hOutputMovie, buffer, len))
{
 loggerVpr.log("\nUnable to write data to file with fd %d", hOutputMovie);
 return false;
}

References

fcntl.h
Suse OS
AIX OS

Functions with a Variable Argument List II

2008-01-16T17:11:00.000-08:00

After using functions with a variable argument list to implement a simple logging sollution I found out that those functions do a great job also in the exception handling.
When you want to throw an exception you usually implement a utility method like this:

void throwByName(JNIEnv *env, const char *name, const char *msg)
{
env->ExceptionDescribe();
env->ExceptionClear();

jclass cls = env->FindClass(name);
/* if cls is NULL, an exception has already been thrown */
if (cls != NULL)
{
env->ThrowNew(cls, msg);
}
/* free the local ref */
env->DeleteLocalRef(cls);
}

This method can be invoked in such a way:

char buffer[128];
throwByName(env, "com/exceptions/MyException", buffer);

The problem is that if you have a lot of methods and exception condition you have to copy/paste every time the exception class that is inconvenient. Also after the first refactoring you'll probably have to change the exception class on so much places.
To avoid those problems you can implement utility methods for all the exceptions that are used (or maybe the most often used) to store the exception class:

void throwMyException(JNIEnv * env, const char * errorMessage, ...)
{
char buffer[STACK_TRACE_SIZE];
va_list args;
va_start(args, errorMessage);
vsnprintf(buffer, sizeof(buffer), errorMessage, args);

throwByName(env, "com/exceptions/MyException", buffer);
}

The benefit is that you can throw that particular exception very easy and without much copy/paste:

throwMyException(env, "Error finishing encryption!");

You can also take a look at this post for more tips&trick for JNI.

Functions with a Variable Argument List

2008-01-16T08:12:00.000-08:00

My current project is a JNI application - invokes C++ code from Java. I began implementation of a Logger component for the C++ part. As usually I wanted to integrate it very easy with just replacing the previous printing to stdout (printf) with a utility method (log) eg:
replace: printf("Error = %d (%s)", error, errorMessage);
with : log("Error = %d (%s)", error, errorMessage);
The signature of the log method is: log(const char * message, ...) that should print the log message to a file with: fprintf (file, messageString).

The big problem turned out to be - how to pass all the arguments from log to fprintf - both methods with variable argument list ...

Some ideas:

All the tutorials learn you how to iterate those variable arguments and manipulate them separately - time consuming
Overloade operator << - a lot of refactoring required.
A cool idea was to redirect the System.out from Java to a file with System.setOut(printStream) - stdout is not the same as System.out

A possible sollution is to redirect the stdout from the C++ code to a file and skip the whole Logger component - the printf will append to the log file:
std::freopen(LOG_FILE, "a", stdout);
The big side effect is that the Java System.out stream is also redirected and writes to the file.

And the winner is:

#include "stdio.h"
#include "stdarg.h"
log(const char * message, ...) {
char buffer[512]
va_list args;
va_start(args, message);
// Returns the size of the created message
vsnprintf(buffer, sizeof(buffer), message, args);
    fprintf (file, buffer);
}

vsnprintf formats the message with the argument list and writes it to the buffer that is easily logged.

Clean sollution but far from my Java-stuffed brain.


A solution for exception handling in JNI is in: Functions with a Variable Argument List II
You can also take a look at this post for more tips&trick for JNI.

Java Native Interface (JNI) Tutorial - Hell on Stage

2007-12-13T06:19:00.000-08:00

Intro

There are so many tutorials about using JNI to access C/C++ code from Java so I don't want to write the next ... but to share some tips & tricks with a very short overview.

Java and C together! Why?

Implement time-critical code
Access legacy code or code libraries from Java programs
Need of platform-dependent features not supported in Java (not covered)

Calling C/C++ code from Java programs

Six steps to do this with Java Native Interface:

Write Java code
public class Sample{ public native String stringMethod(String text); public static void main(String[] args) { System.loadLibrary("sample"); Sample sample = new Sample(); String text = sample.stringMethod("java"); System.out.println("stringMethod: " + text) }
Compile the Java code
javac Sample.java
Generate C/C++ header file
Generate Sample.h from the java class file: $javah -jni Sample. The result uses a pointer to a table of function pointers (JNIEnv parameter) and a parameter refers to the current invoking Java object or this pointer (jobject) :
#include extern "C" { JNIEXPORT jstring JNICALL Java_Sample_stringMethod (JNIEnv *, jobject, jstring); }
Write C/C++ code
Implement the methods from the generated header files:
JNIEXPORT jstring JNICALL Java_Sample_stringMethod(JNIEnv *env, jobject obj, jstring string) { const char *str = env->GetStringUTFChars(string, 0); char cap[128]; strcpy(cap, str); env->ReleaseStringUTFChars(string, str); return env->NewStringUTF(strupr(cap)); }
Create shared library file
The naming convension for the output shared library is different depending on the OS and must be followed: sample.dll (Windows) or libsample.so (*NIX).
$g++ -I /usr/lib/jvm/java-1.5.0-sun/include sample.cpp -L ~/resources/verimatrix -shared -o libsample.so -lcrypto
Run the Java application
java Sample

Access shared libraries

Prerequisites
Usually you have some shared libraries that provide some functionality. For example two dynamic linux shared libraries (libvpr.so and libringdll.so).
Problems

Difference between the provided method names in the libraries and the naming convension used by JNI - you cannot invoke one of the methods directly unless it is designed with JNI in mind. JNI requires some prefixes as shown above with the method name "Java_Sample_stringMethod"
Not all JNI data types can be mapped to C/C++ data types and vice versa:
jint == int, jbyte == jbyte
jbyteArray != byte *, jstring != string, jclass != class
Freeing Native Resources - some resources cannot be freed by the GC like strings and global references.

Access C/C++ methods provided by the libraries
The solution that handles the above problems is to implement a proxy library that follows the JNI naming convention and proxies all method calls to the functional shared library:

Implement Java code with all required native methods:
private native boolean VprEncryptMovie(EncryptMovieSettings encryptionSettings);
Design C/C++ shared library that implement the interface of the methods as proxy:
jboolean Java_jni_VerimatrixClient_VprEncryptMovie(EncryptMovieSettings jobject);
Implement data conversion from the JNI data types to standard C/C++ data types
Proxy all method calls to the corresponding methods in functional libraries
Implement exception handling
jclass newExc=env->FindClass("java/lang/IllegalArgumentException"); env->ThrowNew(newExc, "thrown from C code");

Tips and tricks

Use Apache Harmony VM for trouble shooting or the patched version as described here - it generates a JNI stack trace instead of just crashing the VM as JDK5 or JDM6 eg:

SIGSEGV in VM code. Stack trace: 0: memcpy (??:-1) 1: VprRpcEncryptMovieWithKey(rpc_handle_s_t*, void*, unsigned char*, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, unsigned char*, long, long, long*, long*) (??:-1) 2: VprEncryptMovieWithKey (??:-1) 3: Java_com_minerva_edc_vig_verimatrix_vcas_VprClient_VprEncryptMovieWithKey (/home/emo/perforce/3ML/SB/VideoIngest/src/com/minerva/edc/vig/verimatrix/proxy/VprProxy.cpp:100) 4: 0xA591A50B 5: com/minerva/edc/vig/verimatrix/vcas/VprClient.createEncryptSession(Lcom/minerva/edc/vig/verimatrix/vcas/model/VprServerRegisterResponse;Lcom/minerva/edc/vig/verimatrix/vcas/model/EncryptMovieSettings;Lcom/minerva/edc/vig/verimatrix/vcas/model/MovieInfo;Lcom/minerva/edc/vig/core/data/DTO;)J (VprClient.java:73) ... 23: _start (/usr/src/packages/BUILD/glibc-2.3/csu/../sysdeps/i386/elf/start.S:105)
Eclipse plugin for C/C++ development - CDT
Do not store local variables in the C++ classes if more that one simultaneous client to the proxy is planned - only one instance of this class from the library is loaded by the JVM.
You can implement a Java class to pass multiple parameters to a JNI function and extract them with reflection but remmember that every reflection call has big performance hit. Also if this input parameter is "converted" into a C/C++ structure or class, the constructed objects cannot be cached (see the previos tip). If the parametes are not so many - pass them all.
A Java class can be returned as out parameter by reference. It is better to create the instance in the Java and just fill the properties in the C++ proxy. This eliminates the need to call the constructor of the response object and to hardcode its package and class name in the proxy class. Only the method names are required.

A setter method can be invoked in such a way:

// Create the encryption settings class jclass vprResponseClass = env->GetObjectClass(vprResponseObj); // Allocates memory for an array of one long long output parameter jvalue* args = (jvalue*)malloc(sizeof(jlong)); way: // Create the encryption settings class jclass vprResponseClass = env->GetObjectClass(vprResponseObj); // Allocates memory for an array of one long long output parameter jvalue* args = (jvalue*)malloc(sizeof(jlong)); args[0].j=input_file_size; jmethodID setInputFileSizeID =env->GetMethodID(vprResponseClass,"setInputFileSize", "(J)V"); env->CallLongMethodA(vprResponseObj, setInputFileSizeID, args); // Cleanup free((jvalue*) args);

A thrown exception in a C++ method do not stops its execution so all allocated resources in the method can be freed first before the return clause.

Passing Invalid Arguments to JNI Functions - most often JVM crashes. Use JVM option -Xcheck:jni to detect errors like passing passing NULL or (jobject)0xFFFFFFFF in place of reference. This option degrates the performance.

Troubleshooting

java.lang.UnsatisfiedLinkError: no sample in java.library.path
The library name in Java (System.loadLibrary)is wrong - chech the prefix or extension
Could not find the library - set the directory locations as a JVM parameter: -Djava.library.path=/home/lib:/opt/lib/

java.lang.UnsatisfiedLinkError: /libverimatrixproxy.so: libvpr.so: cannot open shared object file: No such file or directory

A library dependency cannot be found. Add the location of the missing library e.g. ~/lib/ in "/etc/ld.so.conf". Then reconfigure linker bindings $ldconfig

java.lang.UnsatisfiedLinkError: /libverimatrix.so: /libverimatrix.so: undefined symbol: VprDestroyContext
Check the shared library for all undefined symbols in the list of its dependences
$ldd -d ~/lib/libverimatrix.so
Place the missing library that provides those methods, structres, enumerations or ... in the libraries dependency search path.

java.lang.UnsatisfiedLinkError: /libverimatrix.so: Can't load IA 32-bit .so on a IA 32-bit platform
Bug in JDK5 - wrong error message. Use another VM (JDK6, Harmony) to get the correct stack trace.

[Too many errors, abort]
Infinite error message dump:
Many times: [error occurred during error reporting, step 270, id 0xb]
Then infinite: [Too many errors, abort]
This occurs when the JDK detects an error in called C++ code but does not crash imediate. See the first tip about the Apache Harmony JVM.

error: redefinition of ‘struct EncryptMovieSettings’
This type of C++ compile error may happpen if a header file is called or included more than once. Try to precede and end it like that to avoid that:

#ifndef __yourheader_h
#define __yourheader_h (1)
// Put here the body of your ".h" file including the class
#endif

Debug JNI

Integrated Debugger for JNI Environments
Eclipse version 3.2 (not a newer) can be used to debug both the Java and C++ code of JNI. Apache Harmony JVM provides interface for agent which could manage and handle events in debug session.
Synchronized Sollution
Solution with two debuggers that attach to a running Java or C++ process and cooperate - a Java debugger eg. Eclipse and a C/C++ debugger eg. Insight - GUI frondend to GDB

References

Java Native Interface 1999.pdf
Java programming with JNI.pdf

GMAIL over IMAP! The battle is over!

2007-10-25T01:57:00.000-07:00

Finally GMAIL will have IMAP support!
I'm so tired of this battle with the POP3 access. And the best is that all win :)

I started using GMAIL some years ago but lately the need to check all my three accounts through the web interface with the enormous loading time almost made me give up these mails.
I tried the POP forwarding to another mail account with IMAP support but could not find such proper mail service.

The winner was AOL mail with unlimited space but after some doubt I found that the mails are kept for 28 days only before complete deletion. At second place was www.bluebottle.com with 250MB space but I couldn't register receiving some error with wrong message:There have been too many signups from your IP address in the last 24 hours. All other mail servers offered for free only no more than 20-30MB of space ~= 5-6 mails with attachments and I returned to the old painful POP3 access.

Tools for every day from JDK 6

2007-10-09T06:38:00.000-07:00

JDK 6 introduced some usefull tools:

1. JPS - (Also incuded in JDK 5) lists all running JVMs (also those loaded from JNI !!!) with information for them as passed arguments and the  name of the application's main class with its arguments:
$jps -mlvV
3370 sun.tools.jps.Jps -mlvV -Dapplication.home=/usr/lib/jvm/java-6-sun-1.6.0.00 -Xms8m
30707 -Xms512m -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=128m
3327 tcpmon.MainWindow

2. JINFO - prints Java configuration information for a given Java process. The most usefull info as sun.boot.library.path, java.library.path and java.class.path can be filtered:
$jinfo 30707| grep -E "(java.class.path|library.path)"

Those tools can be combined to expract the most usefull information for all JDK6 in a BASH script ( jpall.sh)  skipping the JPS process itself :
#!/bin/bash
for i in $( jps | grep -v Jps | awk -F " " '{print $1}' ) ;
do
   echo
   jinfo $i | grep -E "(java.class.path|library.path)"
   jps -mlvV | grep $i
done

The Five Rules of Compilation

2006-07-10T04:38:00.000-07:00

The complete rules for a compilator based on these three rules:

0. Security: A compiler may not harm any resource or running application.
1. Correctness: A compiler may not harm a source program.
2. Efficiency: A compiler must be fast, except where speed would conflict with the first law.
3. Friendliness: A compiler must assist the user to correct programming errors, as long as such assistance does not conflict with the first and second laws.
4. Informative: A compiler must establish its identiy as the accurate compiler in all cases.
5. Specificness: A compiler must know it is a compiler

Asimov's rules of robotics.

Java interfaces convention

2006-05-31T04:22:00.000-07:00

IPortletDefinition or PortletDefinition.
Reasons to drop the I's

JavaDoc and IDEs like Eclipse differentiate interfaces from classes sufficiently. The Java typing does not also need to be reflected in the type name because our IDEs already give us sufficient visual clues.
Follow the Sun conventions with the goal of minimizing the "impedance mismatch" between our code-base and the JDK, i.e., our APIs should look like they could be in the JDK. This strategy is helpful pedagogically.
Keeping with the norm of other open source and java projects.

Reasons to keep the I's

For the sake of consistency between different versions
To make it easier to tell which files represent java interfaces when viewing files in a list outside of JavaDoc or an IDE like Eclipse, or in Eclipse views which do not illustrate the difference with icons.
To make it easier to identify java interfaces when looking at java code itself.
Having the extra I doesn't present a real burden.

The ideot's mission

2006-05-31T04:10:00.000-07:00

Knowledge is power - learn it and become a powerman.