Programming and Computer tips

EJBs and Servlets

2009-02-13T18:53:00.000-08:00

EJBs and Servlets
Question:
Both Enterprise JavaBeans and Servlets are server side components. What are the major differences between them? Specifically, what is the difference between a Session Bean and a Servlet?

Answer:
Enterprise JavaBeans are components meant to encapsulate business logic. They do not handle presentation and have a precise set of restrictions they must obey. An EJB may not manage threads, access files with the java.io package, have a GUI, accept socket connections, or load native libraries. These restrictions are imposed because EJBs execute inside an EJB container, roughly analogous to the way servlets execute in a servlet container. Where servlets can be used to perform many arbitrary tasks, including the generation of HTML pages, EJBs are used almost entirely as business objects. A session bean represents a client for a session or transaction and an entity bean represents a persistent business object that is usually stored in a database. Unlike servlets, a single session bean may exist per client. A single servlet may serve many clients. A session bean usually mediates client access to entity beans, shielding the client from the details of managing state.

A very common model is for a servlet to act as a client of a session bean. In this multi-tier model, a user talks to a servlet or JavaServer Page through a Web browser. The servlet manages the session, keeping track of a session bean for each of its clients and forwarding transaction requests to the session bean. The session bean may then access any number of entity beans to do things like lookup customer account data. The session bean executes its business logic and returns results to the servlet which then formats a result for the Web browser.

One of the ideas behind EJBs is to componentize server-side business logic and move it closer to the database. EJBs and servlets will typically not run on the same server. A key thing to remember is that both EJBs and servlets must execute inside of a container. For servlets, this is usually a Web server interfaced with a servlet engine. For EJBs, this is an EJB server such as the reference implementation provided with the Java 2 Enterprise Edition SDK.

What to do if Remote JMS Client can not get factory and throw classcastexception?

2008-12-06T16:38:00.000-08:00

What to do if Remote JMS Client can not get factory and throw classcastexception?

First: Add the following property to initialcontext like:

props.put("java.naming.factory.initial", "org.jnp.interfaces.NamingContextFactory");
props.put("java.naming.provider.url", "jnp://localhost:1099");
props.put("java.naming.factory.url.pkgs", "org.jboss.naming:org.jnp.interfaces");

ctx = new InitialContext(props);

Second: Check the version of jar file: jboss-messaging-client. It has to have the same version with jms in jboss.

That's all

JMS on JBoss using EJB3 MDB

2008-09-25T17:14:00.000-07:00

Introduction

This simple JMS application has you submit a form which sends a message to a JBoss queue. A Message-Driven Bean (MDB) will be listening on that queue destination and receives the mssage and processes it. What makes this lesson different than this one is that this MDB example runs in an EJB3 container and the MDB is exposed using a simple annotation. To appreciate this simple example, it will really help to view the 'old way' of having to do things in the link above. With the EJB3 approach there is a lot less configuration. We don not need to configure the jboss-web.xml, destinations-service.xml, ejb-jar.xml, or the jboss.xml.

The Setup

Download the source Download the source above and unzip it into your project directory of choice. For eclipse users the source comes with an eclipse project so you can just import that project file and be good to go. To build using ant, type 'ant all' or if you want to build and deploy to jboss type 'ant deploy' to build and deploy the ear to JBoss.

Java5 The Java5 JDK needs to be set as your environment. JAVA_HOME env variable needs to be defined pointing to yth the JDK home directory.

JBoss 4.2+ - Download JBoss 4.2 here http://labs.jboss.com/portal/jbossas/download. (This should also run on 4.0.5 if it's patched to support EJB3 using the JEMS installer.)

Running the example If you've built the project usingg ant (ant deploy), the ear will be deployed to jboss and you simply go to your jboss home dir/bin and start jboss with run.sh (Nix flavors) or run.bat (Windows). If you want to just see the app run you can download the ear above and put the ear into your jboss/default/deploy directory. Once jboss is started, go the following url: http://localhost:8080/rr-jms

LongProcessMessageBean

This is our Message Driven Bean. Notice the EJB3 annotations. The annotation tells the container to deploy this as an Message Driven Bean and it also creates the queue based off our annotation. We are also using Resource injection with the @Resource annotation which injects our MessageDrivenContext message which we use if their was some business logic problem and we need to do a rollback in the catch block ( context.setRollbackOnly() .) All this MDB does is mimic calling some business process (doLongProcess). You'll see in your jboss console the println statements when this gets called. (Of course in real life you'd be using Logging and not stupid println statements.)


import javax.annotation.Resource;
import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.ejb.MessageDrivenContext;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.ObjectMessage;

import net.learntechnology.dto.ProcessDTO;
import net.learntechnology.util.LongProcessService;
import net.learntechnology.util.ProcessResult;

@MessageDriven(name="LongProcessMessageBean", activationConfig = {
   @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
   @ActivationConfigProperty(propertyName="destination", propertyValue="queue/examples/OurSampleQueue")

})
public class LongProcessMessageBean implements MessageListener {

   @Resource
   private MessageDrivenContext context;

   public void onMessage(Message message) {
       String name = null;
       try {
           if (message instanceof ObjectMessage) {
               ObjectMessage objMessage = (ObjectMessage) message;
               Object obj = objMessage.getObject();
               if (obj instanceof ProcessDTO) {
                   name = ((ProcessDTO)obj).getName();
                   System.out.println("****************************************************");
                   System.out.println("LongProcessMessageBean.onMessage(): Received message. NAME: "+name);
                   System.out.println("****************************************************");
                   System.out.println("Now calling LongProcessService.doLongProcess");

                   ProcessResult result = LongProcessService.doLongProcess((ProcessDTO)obj);
                } else {
                   System.err.println("Expecting ProcessDTO in Message");
               }
           } else {
               System.err.println("Expecting Object Message");
           }
           System.out.println("*******************************************");
           System.out.println("Leaving LongProcessMessageBean.onMessage(). NAME: "+name );
           System.out.println("*******************************************");
       } catch (Throwable t) {
           t.printStackTrace();
           context.setRollbackOnly();
       }
   }    
}

index.jsp form

A simple JSP page. All it does is submit some text to our Servlet (AppServlet).


   Your name: lt;input type="text" name="name" size="40">

AppServlet

Our AppServlet takes the request of text submitted from index.jsp and passes it to our JmsProducer. Our JmsProducer will need to know the jndi names of the connection factory and the queue name, so we are defining them here as well and passing them to our JmsProducer class.


public class AppServlet extends HttpServlet {
   private static final String QUEUE_CONNECTION_FACTORY = "ConnectionFactory";
   private static final String LONG_PROCESS_QUEUE = "queue/examples/OurSampleQueue";

   public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
       doPost(request, response);
   }

   public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
       System.out.println("in doPost of AppServlet..");
       String name = request.getParameter("name");
       System.out.println("name: "+name );
       ProcessDTO processDTO = new ProcessDTO();
       processDTO.setName(name);
       JmsProducer.sendMessage(processDTO, QUEUE_CONNECTION_FACTORY, LONG_PROCESS_QUEUE);
       request.setAttribute("started",true);
       RequestDispatcher dispatcher =  getServletContext().getRequestDispatcher("/index.jsp");
       dispatcher.forward(request, response);
   }
}

JmsProducer

This is our JMS client code that sends the message to the queue. We pushed the Connection lookup and the Destination lookup into another class called 'ServiceLocator' which is shown next.


public class JmsProducer {
   private JmsProducer() {}
   public static void sendMessage(Serializable payload, String connectionFactoryJndiName,
       String destinationJndiName) throws JmsProducerException {
       try {
           ConnectionFactory connectionFactory = null;
           Connection connection = null;
           Session session = null;
           Destination destination = null;
           MessageProducer messageProducer = null;
           ObjectMessage message = null;
           System.out.println("In sendMessage of JmsProducter,
               getting ConnectionFactory for jndi name: "+connectionFactoryJndiName );
           connectionFactory = ServiceLocator.getJmsConnectionFactory(
                                                    connectionFactoryJndiName);

           connection = connectionFactory.createConnection();
           session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
           destination = ServiceLocator.getJmsDestination(destinationJndiName);
           messageProducer = session.createProducer(destination);
           message = session.createObjectMessage(payload);
           messageProducer.send(message);
           System.out.println("Message sent to messageProducer");
           messageProducer.close();
           session.close();
           connection.close();
       } catch (JMSException je) {
           throw new JmsProducerException(je);
       } catch (ServiceLocatorException sle) {
           throw new JmsProducerException(sle);
       }
   }
}

ServiceLocator

This is a simple utility class used to look up our Connection and Destination via a JNDI lookup. In real life we might use a cache as well to store these lookups.


public class ServiceLocator {
  private ServiceLocator() {}
    public static ConnectionFactory getJmsConnectionFactory(String jmsConnectionFactoryJndiName)
            throws ServiceLocatorException {
       ConnectionFactory jmsConnectionFactory = null;
       try {
           Context ctx = new InitialContext();
           jmsConnectionFactory = (ConnectionFactory) ctx.lookup(jmsConnectionFactoryJndiName);
       } catch (ClassCastException cce) {
           throw new ServiceLocatorException(cce);
       } catch (NamingException ne) {
           throw new ServiceLocatorException(ne);
       }
       return jmsConnectionFactory;
   }
   public static Destination getJmsDestination(String jmsDestinationJndiName)
           throws ServiceLocatorException {
       Destination jmsDestination = null;
       try {
           Context ctx = new InitialContext();
           jmsDestination = (Destination) ctx.lookup(jmsDestinationJndiName);
       } catch (ClassCastException cce) {
           throw new ServiceLocatorException(cce);
       } catch (NamingException ne) {
           throw new ServiceLocatorException(ne);
       }
       return jmsDestination;
   }
}

You might be wondering how to set the IntialContext with the proper host name to do its lookups. The best place to do that is in a jndi.properties file (which this app uses). Our jndi.properties file looks like this:


#jndi.properties
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
java.naming.provider.url=localhost:1099

application.xml

Necessary for our ear deployment to jboss.



            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
                                http://java.sun.com/xml/ns/j2ee/application_1_4.xsd"
            version="1.4">
   rr-jms-ejb3 Example EAR
  
       
           
               rr-jms-webapp.war
               rr-jms
           
       
       
           conf.jar
       
       
           common.jar
       
  
       
           rr-jms-ejb.jar

PHP - How to parse RSS?

2008-08-03T07:16:00.000-07:00

It is very easy to parse RSS and print out all information of RSS as following
All we need is calling function simplexml_load_string


$rss = '';
$url = "http://www.dantri.com.vn/Sukien.rss";
$rss_file = file_get_contents($url);
if(!$feed = simplexml_load_string($rss_file)){
 die("Cannot load RSS Feed. This application supports RSS 1.0 and 2.0");
}

// print out the title of the feed
echo '';
echo $feed->channel->title;
echo '
';
// check for RSS Version
$items = ($feed['version'] != '') ? $feed->channel->item : $feed->item;
// PRINT OUT ITEMS
/*---------------------------------------------------------------------------------------*/
foreach($items as $item)
{
 echo $item->description.'

';
}
/*---------------------------------------------------------------------------------------*/

How to log sql statement stored in PreparedStatement

2008-05-22T21:17:00.000-07:00



Take the following program.

import java.net.URL;
import java.sql.*;

class JDBCapp  {
  static MyConnection theConn;

  public static void main (String args[]) {
    new JDBCapp().doit();
    }

  public void doit() {
    theConn = new MyConnection();
    theConn.connect("EAS Demo DB V3", "dba", "sql");

PreparedStatement prepstmt;
try {
  prepstmt = theConn.dbConn.prepareStatement
   ("SELECT emp_id FROM employee" );
  prepstmt.execute();
  prepstmt.close();
  }
catch (Exception e) { e.printStackTrace(); }
  theConn.disconnect();
  }
}


class MyConnection {
  Connection dbConn = null;
  void connect(String db, String user, String passw) {
    try {
      Driver d = 
       (Driver)Class.forName("sun.jdbc.odbc.JdbcOdbcDriver").newInstance();
      String URL = "jdbc:odbc:" + db;
      dbConn = DriverManager.getConnection(URL, user, passw);
      java.io.PrintWriter w =
        new java.io.PrintWriter
           (new java.io.OutputStreamWriter(System.out));
      DriverManager.setLogWriter(w);
      }
    catch (Exception e) {
      e.printStackTrace();
      }
    }

  void disconnect() {
    try {
      dbConn.close();
      }
    catch (Exception e) {
      e.printStackTrace();
      }
    }
}

If we use BasicDataSource to getConnection instead of DriverManager,
use BasicDataSource .setLogWriter.

In addition, to log these sql strings to log4j, we will redirect these messages from printwriter to log4j. Use the following code to do this.

http://www.cenqua.com/clover/eg/jboss/report/org/jboss/logging/util/LoggerWriter.html

Show lock on table in oracle and unlock them

2008-05-21T01:35:00.000-07:00

Show lock
=========
select O.object_name,
O.object_type,
S.sid,
S.serial#,
L.oracle_username,
L.os_user_name,
L.locked_mode
from dba_objects O,
v$locked_object L,
v$session S
where O.object_id = L.object_id
and S.sid = L.session_id
=============

Unlock
ALTER SYSTEM KILL SESSION 'sid, serialid';

Angry with entrecard

2008-05-05T23:12:00.000-07:00

I am very angry with entrecard system. They considers my site was spam blog.
They said that because entrecard is their system and they want to do what they want.
It is not acceptable. I will make my own system which can be better than entrecard. If we have more competition, i think they will know where they are in internet.
They delete my site but all their ads is still in my site. What do you think about them? They are all bullshit.

How to use org.apache.commons.httpclient and excute method via proxy

2008-04-12T07:07:00.000-07:00

I try to find out how to use httpclient with proxy and do not want to change java system property. But it seems that we can not find much document about it.
Finally, i have find out the solution:

First: Create httpclient
HttpClient client = new HttpClient();

set proxy
client.getHostConfiguration().setProxy(proxyhost, proxyPort);

//create authenticate
AuthScope auth = new AuthScope(proxyhost,proxyPort, AuthScope.ANY_REALM);
Credentials cred = new UsernamePasswordCredentials("","");
client.getState().setProxyCredentials(auth, cred);
=> now we can connect to internet through proxy.

If you want more information, please post comment. I will answer.

Privacy Policy

2008-03-08T18:41:00.000-08:00

Privacy Policy

The privacy of our visitors to http://thegioitinhoccuatoi.blogspot.com is important to us.

At http://thegioitinhoccuatoi.blogspot.com, we recognize that privacy of your personal information is important. Here is information on what types of personal information we receive and collect when you use visit http://thegioitinhoccuatoi.blogspot.com, and how we safeguard your information. We never sell your personal information to third parties.

Log Files

As with most other websites, we collect and use the data contained in log files. The information in the log files include your IP (internet protocal) address, your ISP (internet service provider, such as AOL or Shaw Cable), the browser you used to visit our site (such as Internet Explorer or Firefox), the time you visited our site and which pages you visited throughout our site.

Cookies and Web Beacons

We do use cookies to store information, such as your personal preferences when you visit our site. This could include only showing you a popup once in your visit, or the ability to login to some of our features, such as forums.

We also use third party advertisements on http://thegioitinhoccuatoi.blogspot.com to support our site. Some of these advertisers may use technology such as cookies and web beacons when they advertise on our site, which will also send these advertisers (such as Google through the Google AdSense program) information including your IP address, your ISP , the browser you used to visit our site, and in some cases, whether you have Flash installed. This is generally used for geotargeting purposes (showing New York real estate ads to someone in New York, for example) or showing certain ads based on specific sites visited (such as showing cooking ads to someone who frequents cooking sites).

You can chose to disable or selectively turn off our cookies or third-party cookies in your browser settings, or by managing preferences in programs such as Norton Internet Security. However, this can affect how you are able to interact with our site as well as other websites. This could include the inability to login to services or programs, such as logging into forums or accounts.
Contact Information

If users have any questions or suggestions regarding our privacy policy, please contact me through my contact form or send me an email to: testoldfriend(at) yahoo (dot) com.

Hibernate, JBoss and auto-increment field

2008-03-07T22:20:00.000-08:00

The element

This is the optional element under element. The element is used to specify the class name to be used to generate the primary key for new record while saving a new record. The element is used to pass the parameter (s) to the class. Here is the example of generator element from our first application:

In this case element do not generate the primary key and it is required to set the primary key value before calling save() method.

Here are the list of some commonly used generators in hibernate:

Generator	Description
increment	It generates identifiers of type long, short or int that are unique only when no other process is inserting data into the same table. It should not the used in the clustered environment.
identity	It supports identity columns in DB2, MySql, MS SQL Server, Sybase and HypersonicSQL. The returned identifier is of type long, short or int.
sequence	The sequence generator uses a sequence in DB2, PostgreSQL, Oracle, SAP DB, McKoi or a generator in Interbase. The returned identifier is of type long, short or int
hilo	The hilo generator uses a hi/lo algorithm to efficiently generate identifiers of type long, short or int, given a table and column (by default hibernate_unique_key and next_hi respectively) as a source of hi values. The hi/lo algorithm generates identifiers that are unique only for a particular database. Do not use this generator with connections enlisted with JTA or with a user-supplied connection.
seqhilo	The seqhilo generator uses a hi/lo algorithm to efficiently generate identifiers of type long, short or int, given a named database sequence.
uuid	The uuid generator uses a 128-bit UUID algorithm to generate identifiers of type string, unique within a network (the IP address is used). The UUID is encoded as a string of hexadecimal digits of length 32.
guid	It uses a database-generated GUID string on MS SQL Server and MySQL.
native	It picks identity, sequence or hilo depending upon the capabilities of the underlying database.
assigned	lets the application to assign an identifier to the object before save() is called. This is the default strategy if no element is specified.
select	retrieves a primary key assigned by a database trigger by selecting the row by some unique key and retrieving the primary key value.
foreign	uses the identifier of another associated object. Usually used in conjunction with a primary key association.

Therefore,
If your application is EJB, WS which create multiple threads, you should not use class=increment. Instead, you should use class=sequence or native. Or you will
encounter the problem: unique constrain violation.

Java: how to find the absolute path with white space to the resource ?

2008-02-15T23:27:00.000-08:00

For example: our program contains the resource A and class MyClass


    src
       main
          java
             MyClass
       resource
          A

When we package our program, A will be at root.
To get the absolute path to resource(for example when we want to do unit test
and load those resource for test), we will call
String path = MyClass.getClass.GetResource("/A").toURI().getPath();
And now we have the absolute path to resource.

JMock and how to mock static function and final class?

2008-02-15T22:46:00.000-08:00

Unit test is one of the "must" task when we develop software. One of the most technology we usually used is mock. As i already said in one of my previous post,
mock is the powerful technology but they have some limitation. This limitation is depend on which type of mock object( JMock, AMock, NMock ...) or the environment.
For example, if you want to test some DAO class which directly access to DBMS, you can not create mock for all JDBC class to do unit test. And in this case, we also can not do unit test because the test depend on the environment(we have to set up an DBMS just for do unit test). Therefore, we have to chose the most suitable way to test: System test, module test, review code ...
JMock like others mock object, has it own limitation. It can not mock final class, static method. It is because of its implementation. For final class, JMock can not inherit the class to create new mock class ==> can not mock this class.
With static method, because static method can not be overriden, so JMock has no way to create mock object.

So in this case, we have to think about design and test first before code
1. Case: our method use final class.

We will use pattern template method to separate code which contain instances of final class.


class OurClass {
    function myFunc() {
        my code
        ......
        URL url = new URL(href); //Can not mock URL because of final class
        url.openstream(...)
        ....
        my code


    } 
}
Our mission is test this function.

In this case, we create new class calls MockHelperTest and a factory MockHelperFactory



IHelperFactory = interface{
    IHelperTest getMockHelper;
}

MockHelperTest = class implements IHelperTest{
        
    void funcHelper() {
      ///Prepare data , result and code for mock
    }
}

and we change our class to:


class OurClass {
    public IHelperFactory factory = null; //factory can be set by unit test
    
    //If we test this function, we will 
    //pass mock factory to this class and 
    //function will call mock function 
    //instead of real code.
    //If we want to to do real code, 
    //just pass null factory, and it will
    //create real factory and then myFunc 
    //will call funcMockHelper which call 
    //real code:         
    //URL url = new URL(href); //Can not mock URL because of final class
    //url.openstream(...)

    OurClass(IHelperFactory  mockFactory) {
         if mockFactory == null ) {
              factory = new RealFactory();
         } else {
              factory = mockFactory;
         }
    }

    function myFunc() {
        
        my code
        ......
        IHelperTest helper = factory.funcHelper().getMockHelper();        
        helper.funcMockHelper();
        ....
        my code
    } 
}

And now, we can easily replace the code which can not mock which the mock code.
The same pattern for static function.

Conclusion: Think about test + design first and then code.

NUnit === How to and FAQ

2008-02-03T09:31:00.000-08:00

NUnit goal:

+ NUnit is a unit-testing framework for all .Net languages. Initially ported from JUnit, the current production release, version 2.4, is the fifth major release of this xUnit based unit testing tool for Microsoft .NET. It is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. NUnit brings xUnit to all .NET languages
Homepage: http://www.nunit.org/index.php

+ Why do we use NUnit?
Because it makes us do unit test very easy and very efficiently

+ How to use NUnit:

1.We have to import library:
using System;
using NUnit.Framework;
2. Declare[TestFixture] attribute to class
3. Declare [SetUp] attribute for setup function. This function will be called before any call to tested function
4. Run NUnit test GUI to test assemblies

+ Some functions need for test:
Assert.AreEqual: raise exception if two object are not the same
Assert.IsTrue: raise exception if condition is false
Assert.IsFalse: raise exception if condition is true
Assert.Fail: Raise exception to notify test case fail.
....

+ How do I implement a test case for a thrown exception?
Catch the exception and if it isn't thrown call the Fail method. Fail signals the failure of a test case. Here is an example:


    public void TestArgumentOutOfRangeException() {
        ArrayList l = new ArrayList(10)
        try {
             Object o= l[l.Count];
             Fail("Should raise an ArgumentOutOfRangeException");
        } catch (ArgumentOutOfRangeException) {
        }
    }

or use the ExceptionTestCase as follows.
1) make your TestCase class a subclass of ExceptionTestCase.
2) write the test ignoring exceptions


    public void TestArgumentOutOfRangeException() {
        ArrayList l= new ArrayList(10);
        l[l.Count];
    }

3) create the TestCase:


    Test t= new ExceptionTestCase("TestArgumentOutOfRangeException", typeof(ArgumentOutOfRangeException))

Looking at this again, the first way is simpler. Sigh...

Another way:
Declare:


[Test]
[[ExpectedException( typeof( AException ) )]
void testFunction() {
...............
};

+ How do I organize my Test Cases?
Here is one way:

1. create a test namespace for each of your application namespaces. For example, for a namespace MyApp.Util define MyApp.UtilTest. Put all the fixtures for the Util namespace into this namespace.
2. in MyApp.UtilTest define a class which creates a suite with all the tests in this namespace. To do so define a class AllTests which includes a single static Suite property. Here is an example:


       public static ITest Suite {
         get {
           TestSuite suite= new TestSuite();
           suite.AddTest(Fixture1.Suite);
           suite.AddTest(Fixture2.Suite);
           return suite;
         }
       }

3. define similar AllTests classes that create higher level suites containing the suites from other test packages.

When the fixtures are in a separate test assembly the test cases don't have access to the methods and fields with internal visibility. A variation of the above convention is to put all fixtures into the application assembly itself. This gives the fixtures access to all the assembly visible methods and fields. To separate the fixture classes from the production classes put them into a separate directory that you then add to the project. This makes it easy to ship the production classes independent of the fixtures.

How do I run setup code once for all my TestCases?
Wrap the top level suite in a subclass of TestSetup. Here is a sample AllTests.Suite method:


    protected class WrappedTestSetup: TestSetup {
      public WrappedTestSetup(ITest test) : base(test) {} 
      protected override void SetUp() {
        OneTimeSetUp();
       }
      };

    public static ITest Suite {
      get {
        TestSuite suite= new TestSuite();
        ...add your tests and suites here...
        TestSetup wrapper= new WrappedTestSetup(suite);
        return wrapper;
      }
    }

+ I want to debug when a test fails
Start the test runner under the debugger and configure the debugger so that it catches the NUnit.Framework.AssertionFailedError. How you do this depends on the used IDE. Most debuggers support to stop the program when a specific exception is fired. Notice, that this will only break into the debugger when an "anticipated" assertion failed error occurs.

Steps:
1. Start C# IDE ( Visual .net ide)
2. Build project + test in debug mode.
3. Configure NUnit GUI to support VS.NET project
3. Add debug assembly to NUnit GUI.
4. Put breakpoint to where you want to debug
Finished.

Sample:


namespace NUnit.Samples.Money 
{
 using System;
 using NUnit.Framework;
 /// 
 /// 
 /// 

 /// 
 [TestFixture]
 public class MoneyTest 
 {
  private Money f12CHF;
  private Money f14CHF;
  private Money f7USD;
  private Money f21USD;
        
  private MoneyBag fMB1;
  private MoneyBag fMB2;

  /// 
  /// 
  /// 

  /// 
  [SetUp]
  protected void SetUp() 
  {
   f12CHF= new Money(12, "CHF");
   f14CHF= new Money(14, "CHF");
   f7USD= new Money( 7, "USD");
   f21USD= new Money(21, "USD");

   fMB1= new MoneyBag(f12CHF, f7USD);
   fMB2= new MoneyBag(f14CHF, f21USD);
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagMultiply() 
  {
   // {[12 CHF][7 USD]} *2 == {[24 CHF][14 USD]}
   Money[] bag = { new Money(24, "CHF"), new Money(14, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Multiply(2));
   Assert.AreEqual(fMB1, fMB1.Multiply(1));
   Assert.IsTrue(fMB1.Multiply(0).IsZero);
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagNegate() 
  {
   // {[12 CHF][7 USD]} negate == {[-12 CHF][-7 USD]}
   Money[] bag= { new Money(-12, "CHF"), new Money(-7, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Negate());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagSimpleAdd() 
  {
   // {[12 CHF][7 USD]} + [14 CHF] == {[26 CHF][7 USD]}
   Money[] bag= { new Money(26, "CHF"), new Money(7, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Add(f14CHF));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagSubtract() 
  {
   // {[12 CHF][7 USD]} - {[14 CHF][21 USD] == {[-2 CHF][-14 USD]}
   Money[] bag= { new Money(-2, "CHF"), new Money(-14, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Subtract(fMB2));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void BagSumAdd() 
  {
   // {[12 CHF][7 USD]} + {[14 CHF][21 USD]} == {[26 CHF][28 USD]}
   Money[] bag= { new Money(26, "CHF"), new Money(28, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, fMB1.Add(fMB2));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void IsZero() 
  {
   Assert.IsTrue(fMB1.Subtract(fMB1).IsZero);

   Money[] bag = { new Money(0, "CHF"), new Money(0, "USD") };
   Assert.IsTrue(new MoneyBag(bag).IsZero);
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MixedSimpleAdd() 
  {
   // [12 CHF] + [7 USD] == {[12 CHF][7 USD]}
   Money[] bag= { f12CHF, f7USD };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, f12CHF.Add(f7USD));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MoneyBagEquals() 
  {
   //NOTE: Normally we use Assert.AreEqual to test whether two
   // objects are equal. But here we are testing the MoneyBag.Equals()
   // method itself, so using AreEqual would not serve the purpose.
   Assert.IsFalse(fMB1.Equals(null)); 

   Assert.IsTrue(fMB1.Equals( fMB1 ));
   MoneyBag equal= new MoneyBag(new Money(12, "CHF"), new Money(7, "USD"));
   Assert.IsTrue(fMB1.Equals(equal));
   Assert.IsTrue(!fMB1.Equals(f12CHF));
   Assert.IsTrue(!f12CHF.Equals(fMB1));
   Assert.IsTrue(!fMB1.Equals(fMB2));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MoneyBagHash() 
  {
   MoneyBag equal= new MoneyBag(new Money(12, "CHF"), new Money(7, "USD"));
   Assert.AreEqual(fMB1.GetHashCode(), equal.GetHashCode());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MoneyEquals() 
  {
   //NOTE: Normally we use Assert.AreEqual to test whether two
   // objects are equal. But here we are testing the MoneyBag.Equals()
   // method itself, so using AreEqual would not serve the purpose.
   Assert.IsFalse(f12CHF.Equals(null)); 
   Money equalMoney= new Money(12, "CHF");
   Assert.IsTrue(f12CHF.Equals( f12CHF ));
   Assert.IsTrue(f12CHF.Equals( equalMoney ));
   Assert.IsFalse(f12CHF.Equals(f14CHF));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void MoneyHash() 
  {
   Assert.IsFalse(f12CHF.Equals(null)); 
   Money equal= new Money(12, "CHF");
   Assert.AreEqual(f12CHF.GetHashCode(), equal.GetHashCode());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Normalize() 
  {
   Money[] bag= { new Money(26, "CHF"), new Money(28, "CHF"), new Money(6, "CHF") };
   MoneyBag moneyBag= new MoneyBag(bag);
   Money[] expected = { new Money(60, "CHF") };
   // note: expected is still a MoneyBag
   MoneyBag expectedBag= new MoneyBag(expected);
   Assert.AreEqual(expectedBag, moneyBag);
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Normalize2() 
  {
   // {[12 CHF][7 USD]} - [12 CHF] == [7 USD]
   Money expected= new Money(7, "USD");
   Assert.AreEqual(expected, fMB1.Subtract(f12CHF));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Normalize3() 
  {
   // {[12 CHF][7 USD]} - {[12 CHF][3 USD]} == [4 USD]
   Money[] s1 = { new Money(12, "CHF"), new Money(3, "USD") };
   MoneyBag ms1= new MoneyBag(s1);
   Money expected= new Money(4, "USD");
   Assert.AreEqual(expected, fMB1.Subtract(ms1));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Normalize4() 
  {
   // [12 CHF] - {[12 CHF][3 USD]} == [-3 USD]
   Money[] s1 = { new Money(12, "CHF"), new Money(3, "USD") };
   MoneyBag ms1= new MoneyBag(s1);
   Money expected= new Money(-3, "USD");
   Assert.AreEqual(expected, f12CHF.Subtract(ms1));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void Print() 
  {
   Assert.AreEqual("[12 CHF]", f12CHF.ToString());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleAdd() 
  {
   // [12 CHF] + [14 CHF] == [26 CHF]
   Money expected= new Money(26, "CHF");
   Assert.AreEqual(expected, f12CHF.Add(f14CHF));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleBagAdd() 
  {
   // [14 CHF] + {[12 CHF][7 USD]} == {[26 CHF][7 USD]}
   Money[] bag= { new Money(26, "CHF"), new Money(7, "USD") };
   MoneyBag expected= new MoneyBag(bag);
   Assert.AreEqual(expected, f14CHF.Add(fMB1));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleMultiply() 
  {
   // [14 CHF] *2 == [28 CHF]
   Money expected= new Money(28, "CHF");
   Assert.AreEqual(expected, f14CHF.Multiply(2));
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleNegate() 
  {
   // [14 CHF] negate == [-14 CHF]
   Money expected= new Money(-14, "CHF");
   Assert.AreEqual(expected, f14CHF.Negate());
  }

  /// 
  /// 
  /// 

  /// 
  [Test]
  public void SimpleSubtract() 
  {
   // [14 CHF] - [12 CHF] == [2 CHF]
   Money expected= new Money(2, "CHF");
   Assert.AreEqual(expected, f14CHF.Subtract(f12CHF));
  }
 }
}

How to use interface correctly?

2008-01-21T21:59:00.000-08:00

A: IparentInterface;
B: IchildInterface;

This code will generate error:

B: = IchildInterface (A); ==> will generate expected error whenever we access to B. At the time we down-cast A, we do not have exception.

The correct code should be:

A.QueryInterface (IchildInterface, B);

So the lesson here: Always use QueryInterface when you want to access other inteface of the same object. Do not cast Interface except you really know what you do.

Delphi Memory Leak

2008-01-21T21:47:00.000-08:00

I have a lession learn today when I work with delphi. We should avoid using Tlist to store object which implemented any interfaces.

We should use TinterfaceList instead of Tlist. The reason is: Tlist use pointer to reference to object. When we get an object and from that object we get one of its interfaces. This interface is not referenced by anyone (although Tlist use pointer to point to the object) .Therefor when go out of variable scope, this interface is freed automatically.

Hereunder is the sample explaining the lession.


InterfaceA = Iinterface 
            Procedure methodA;
End;


ClassA = class(TinterfaceObject, InterfaceA)
Procedure methodA;
End;


Var 
A: TOBbject
List: Tlist;
Begin
List := Tlist.Create;
List.Add(A);
methodX;
methodY; ==> Exception occur here.
End.

Procedure methodX;
Var
   X: InterfaceA;
   Obj: ClassA;
Begin
   Obj := Tlist.Items[0];
  Obj.QueryInterface(InterfaceA, X); ==> Get interface here. Reference count increased to 1
End; ==> Interface will release reference count. Reference count = 0 ==> Obj is freed.
 
Procedure methodY;
Begin
Obj := Tlist.Items[0];
            Obj.methodA; ç Exception because of object is freed.
End;


Hope this lession lean can help you avoid unexpected exception.

Does java have memory leak?

2008-01-21T21:27:00.000-08:00

According to the IBM's artical, JAVA also has problems of memory leak event though they have its own memory garbage collector. The main reason is that: design is not good.

I. How memory garbage collector work?

The job of the garbage collector is to find objects that are no longer needed by an application and to remove them when they can no longer be accessed or referenced. The garbage collector starts at the root nodes, classes that persist throughout the life of a Java application, and sweeps though all of the nodes that are referenced. As it traverses the nodes, it keeps track of which objects are actively being referenced. Any classes that are no longer being referenced are then eligible to be garbage collected. The memory resources used by these objects can be returned to the Java virtual machine (JVM) when the objects are deleted.

II. How it leaks?

"An object is only counted as being unused when it is no longer referenced" ==> The garbage collector can not do its job if an object is referenced by others object for a very long time even though it is no longer needed. It can lead to a terrible problems: out of memory.

Example:

Object A = new Object; //A exists until program terminate.
During program life cycle, an object B is created and referenced by A
Because programmer believe in garbage collector, so they do not free memory manually,
but let garbage collector do it for him ==> B exist until A freed.
If program create many and many B for its life ==> all B will exist too ==> cause out of memory.

It is the obvious example of problem of memory leak. So we do not believe in Garbage Collector 100%. Let free object manually by assigned unused object to nil.

III.Preventing memory leaks

You can prevent memory leaks by watching for some common problems. Collection classes, such as hashtables and vectors, are common places to find the cause of a memory leak. This is particularly true if the class has been declared static and exists for the life of the application.

Another common problem occurs when you register a class as an event listener without bothering to unregister when the class is no longer needed. Also, many times member variables of a class that point to other classes simply need to be set to null at the appropriate time.

Hope i can help you in avoid memory leak in java.

How to get the path and filename of executable file of current process (or running process)?

2008-01-19T04:57:00.000-08:00

How to get the path + file name of the current process or the running process?
There are two way to this:

I will illustrate idea in delphi

1. You can call the API function: GetProcessImageFileName

Declare prototype:

interface
function GetProcessImageFileNameA(hProcess: THandle; lpImageFileName: LPSTR;
nSize: DWORD): DWORD; stdcall;
{$EXTERNALSYM GetProcessImageFileNameA}
function GetProcessImageFileNameW(hProcess: THANDLE; lpImageFileName: LPWSTR;
nSize: DWORD): DWORD; stdcall;
{$EXTERNALSYM GetProcessImageFileNameW}
function GetProcessImageFileName(hProcess: THANDLE; lpImageFileName: LPTSTR;
nSize: DWORD): DWORD; stdcall;
{$EXTERNALSYM GetProcessImageFileName}

implementation

function GetProcessImageFileNameA(hProcess: THandle; lpImageFileName: LPSTR;
nSize: DWORD): DWORD; stdcall; external 'Psapi.dll' name 'GetProcessImageFileNameA';
function GetProcessImageFileNameW(hProcess: THANDLE; lpImageFileName: LPWSTR;
nSize: DWORD): DWORD; stdcall; external 'Psapi.dll' name 'GetProcessImageFileNameW';
function GetProcessImageFileName(hProcess: THANDLE; lpImageFileName: LPTSTR;
nSize: DWORD): DWORD; stdcall; external 'Psapi.dll' name 'GetProcessImageFileNameA';

Calling function:

+ GetWindowThreadProcessId(WindowHandle, @processID); ==> get processid
+ processHandle := OpenProcess(PROCESS_TERMINATE or PROCESS_QUERY_INFORMATION,
False, processID); ==> Get process handler
+ GetProcessImageFileName(processHandle , bufferout, lenghtofbufferout)

2. The second way:

Call function GetModuleFileName with first parameter nil
+ GetModuleFileName(0, Buffer, 1024); <== Buffer contain the current path of current process.

Delphi --- problem when using function RegisterClass

2007-12-30T04:39:00.000-08:00

Delphi has a feature: serialize/deserialize object. It means you can save/load object at runtime.
To do this, your classes have to inherit from TPersistent and you have to register this class to system. To register a class you can you function: RegisterClass. After registering successfully, you can read or write object to the stream. But there are problems: you can not know if class is registered successfully till you call function: getclass or findclass, and you can not know why it is not succeeded. It takes me 1 day to find why it can not register. All i have to do is debug delphi system to find out the reason. And i want to share my experience when i face this problem. The reason is: the class belongs to a group. But this group is deactivated when delphi create group. So this class can not be registered. All we have to do in this case is activate group again by calling function activategroup(class belong to group). Hope i can help you to save time when solve this problem

HTML Parser -- A good html parser.

2007-12-30T04:25:00.000-08:00

I have found many parser used in my projects. By asking google, we can find some parsers. But after try many parsers, i can tell you that HTMLPaser is the one should be considered.
Website: htmlparser.sourceforge.net
HTMLParser is a parser which can help you parser html file. It has 2 features which i like very much
1. Extracter: It can help us extract information from html file very easily with many filters.
All we have to do is call parser.parse(Filter). A filter can be an individual filter or it can be predicate which combine many filter(and , or , not, user-defined filter) to help you extract as much as possible information.
2. Visitor pattern: With visitor pattern you can do what you want with a tag when parser traverse to that tag. You can change the attribute of node, get information, ... whatever you want.
Besides, htmlparser is well-structured, so it can speed-up coding, maintain easily.

If you want to find a good html parser, why don't give it a try.
PS: You should try html parser version 2.0

Introduction to blogvertise

2007-12-28T07:37:00.000-08:00

Today, i surf net and found an interesting website. You know i love to write blog, and share my experiences i met in my work. It is my hobby. But if i can earn money from those posts,it is really interesting. And Blogvertise is what i want to tell you. You can earn money from your blogs with your posts. The re quirement is simple. You get your email, follow their instruction to write a blog with some criteria. And wait for your money.

Hope it will help you about blogvertise.

Website: Blogvertise

Java: How to load class dynamically (on fly)?

2007-12-09T06:27:00.000-08:00

I face a problem of loading class dynamically in JAVA. How to do this?
After do research in internet with many site, i finally find out the way to do this.
In order to save your time on researching problem, i will show you how to do this.

To understand the framework, we have to know following knowledge:
1. Class loader and how does it work

So how does class loader work:

As shown in Figure 1, the bootstrap class loader (BS) -- Native implemented into JVM -- loads the classes from the JVM, as well as extensions to the JDK. The system class loader (CP) loads all of the classes provided by the CLASSPATH environment variable or passed using the -classpath argument to the java command. Finally we have several additional class loaders, where A1-3 are children of the CP, and B1-2 are children of A3. Every class loader (except BS) has a parent class loader, even if no parent is provided explicitly; in the latter case, the CP is automatically set as the parent.

Rule for class loader:

1. lass loaders are hierarchically organized, where each one has a parent class loader, except the bootstrap class loader (the root).

2. Class loaders should (practically: must) delegate the loading of a class to the parent, but a custom class loader can define for itself when it should do so.

3. A class is defined by its class type and the effective class loader.

4. A class is only loaded once and then cached in the class loader to ensure that the byte code cannot change.

5. Any symbolic links are loaded by the effective class loader (or one of its ancestors), if this is not already done. The JVM can defer this resolution until the class is actually used.

6. An upcast of an instance to another class fails when the class of the instance and the class of the symbolic link do not match (meaning their class loaders do not match).

But another problem is: Classes are loaded by many class loader. How can it distinguish them? JAVA do it by combine qualified name: (Class name, Package name, Class loader) --> With the same classes, but loaded by different class loader, it can not be used together. We can check it easily by using URLClassLoader and sun.misc.Launcher$AppClassLoader(class loader which load main function).

When i work with JAX-WS, i see that we can dynamically add custom handler to WS, and JAX-WS will call our handler. It make me think a lot. How can it do this?
The answer is: JAXWS has its own class loader. And the class loader will find and call our handler.

Now we will make a framework which can allow us load custom handler like JAX-WS at run time.

1. We define a interface for handler. This handler is packaged in jar file.
2. User will define class which implemented the interface handler. The implement class is packaged in other jar file.
3. Our framework will scan those jar file and load all custom handlers, and call it sequentially.

The most difficult part is load all custom handlers and call it at run time.
To solve the problem, we will make our own class loader:


public class CustomLoader extends URLClassLoader
{ 
    static public ClassLoader MainClassLoader = null;
 
    protected CustomLoader (URL[] urls, ClassLoader parent) {
        super(urls, parent);        
    }
    
    public Class loadClass(String name) throws ClassNotFoundException {
        //Get system class loader        
        try { 
         if (MainClassLoader != null) {
          return MainClassLoader.loadClass(name);
         } else {
          return super.loadClass(name);
         }
        } catch (ClassNotFoundException e) {
         return super.loadClass(name);
        }        
    }

    protected Class findClass(String name) throws ClassNotFoundException {
        Class clas = super.findClass(name);
        return clas;
    }
}

The variable MainClassLoader point to sun.misc.Launcher$AppClassLoader( the class loader which call our main function)

How to used this custom class loader?


void main() {
   ClassLoader base = ClassLoader.getSystemClassLoader(); //get the boot trap class loader
   CustomLoader loader = new CustomLoader(urls, base.getParent()); //create our class loader
   loader.MainClassLoader = Thread.currentThread().getContextClassLoader(); //set the  //application class loader to our class loader. We have to do this because when java //compile and link our code, it used class loader: sun.misc.Launcher$AppClassLoader. 

Class handlerClass = loader.loadClass("CustomHandler");
Object handler =  handlerClass.newInstance(); (1)
Handler customHandler = (Handler)handler; (2)
}

(1) --> implement class is loaded by our custom class loader.
(2) --> Handler is loaded by sun.misc.Launcher$AppClassLoader

Thank to this code:

public Class loadClass(String name) throws ClassNotFoundException {
//Get system class loader
try {
if (MainClassLoader != null) {
return MainClassLoader.loadClass(name);
} else {
return super.loadClass(name);
}
} catch (ClassNotFoundException e) {
return super.loadClass(name);
}
}

We delegate loading class to MainClassLoader first: because in linkage phase, MainClassLoader load all symbols in our source code. If we do not do this, we will violate the rule 6. If MainClassLoader does not load them, we will delegate to parent or we violate rule 2. If both do not load class, we can do it by ourselves. But in this case we do not implement this(for simple).

So with our custom class loader, (2) can be used without ClassCastException.

So i hope , you can make a framework easily with the custom class loader as i explain.

Memory Management -- Part 6

2007-11-22T11:32:00.000-08:00

Memory Management

Note the page attributes from the output of the first instance. The functions are marked read-only, as expected. The unshared variable is also marked read-only. This is because Windows NT tries to share the data space also. As described earlier, such pages are marked for copy-on-write, and as soon as the process modifies any location in the page, the process gets a private copy of the page to write to. The other page attributes show that the PTE is valid, the page is a user-mode page, and nobody has modified the page so far.

Now, compare the output from the first instance with the output from the second instance when it loaded the MYDLL.DLL at a base address different from that in the first instance. As expected, the virtual addresses of all the memory sections are different than those for the first instance. The physical addresses are the same except for the physical address of the relocatable function. This demonstrates that the code pages are marked as copy-on-write, and when the loader modifies the code pages while performing relocation, the process gets a private writable copy. Our nonrelocatable function does not need any relocation; hence, the corresponding pages are not modified. The second instance can share these pages with the first instance and hence has the same physical page address.

To cancel out the effects of relocation, the second instance loads MYDLL.DLL at the same base address as that in the first instance. Yup! Now, the virtual address matches the ones from the first instance. Note that the physical address for the relocatable function also matches that in the output from the first instance. The loader need not relocate the function because the DLL is loaded at the preferred base address. This allows more memory sharing and provides optimal performance. It’s reason enough to allocate proper, nonclashing preferred base addresses for your DLLs.

This ideal share-all situation ceases to exist as soon as a process modifies some memory location. Other processes cannot be allowed to view these modifications. Hence, the modifying process gets its own copy of the page The second instance of the sample program demonstrates this by modifying the data variables and a byte at the start of the nonrelocatable function. The output shows that the physical address of the nonrelocatable doesn’t match with the first instance. The nonrelocatable function is not modified by the loader, but it had the same effect on sharing when we modified the function. The shared variable remains a shared variable. Its physical address matches that in the first instance because all the processes accessing a shared variable are allowed to see the modifications made by other processes. But the nonshared variable has a different physical address now. The second instance cannot share the variable with the first instance and gets its own copy. The copy was created by the system page fault handler when we tried to write to a read-only page and the page was also marked for copy-on-write. Note that the page is now marked read-write. Hence, further writes go through without the operating system getting any page faults. Also, note that the modified pages are marked as dirty by the processor.

SWITCHING CONTEXT

As we saw earlier, Windows NT can switch the memory context to another process by setting the appropriate page table directory. The 80386 processor requires that the pointer to the current page table directory be maintained in the CR3 register. Therefore, when the Windows NT scheduler wants to perform a context switch to another process, it simply sets the CR3 register to the page table directory of the concerned process.

Windows NT needs to change only the memory context for some API calls such as VirtualAllocEx(). The VirtualAllocEx() API call allocates memory in the memory space of a process other than the calling process. Other system calls that require memory context switch are ReadProcessMemory() and WriteProcessMemory(). The ReadProcessMemory() and WriteProcessMemory() system calls read and write, respectively, memory blocks from and to a process other than the calling process. These functions are used by debuggers to access the memory of the process being debugged. The subsystem server processes also use these functions to access the client process’s memory. The undocumented KeAttchProcess() function from the NTOSKRNL module switches the memory context to specified process. The undocumented KeDetachProcess() function switches it back. In addition to switching memory context, it also serves as a notion of current process. For example, if you attach to a particular process and create a mutex, it will be created in the context of that process. The prototypes for KeAttachProcess() and KeDetachProcess() are as follows:

NTSTATUS KeAttachProcess(PEB *);

NTSTATUS KeDetachProcess ();

Another place where a call to the KeAttachProcess() function appears is the NtCreateProcess() system call. This system call is executed in the context of the parent process. As a part of this system call, Windows NT needs to map the system DLL (NTDLL.DLL) in the child process’s address space. Windows NT achieves this by calling KeAttachProcess() to switch the memory context to the child process. After mapping the DLL, Windows NT switches back to the parent process’s memory context by calling the KeDetachProcess() function.

The following sample demonstrates how you can use the KeAttachProcess() and KeDetachProcess() functions. The sample prints the page directories for all the processes running in the system. The complete source code is not included. Only the relevant portion of the code is given. Because these functions can be called only from a device driver, we have written a device driver and provided an IOCTL that demonstrates the use of this function. We are giving the function that is called in response to DeviceIoControl from the application. Also, the output of the program is shown in kernel mode debugger’s window (such as SoftICE). Getting the information back to the application is left as an exercise for the reader.

void DisplayPageDirectory(void *Peb)

{

unsigned int *PageDirectory =

(unsigned int *)0xC0300000;

int i;

int ctr=0;

KeAttachProcess(Peb);

for (i = 0; i < 1024; i++) {

if (PageDirectory[i]&0x01) {

if ((ctr%8) == 0)

DbgPrint(" \n");

DbgPrint("%08x ", PageDirectory[i]&0xFFFFF000);

ctr++;

}

}

DbgPrint("\n\n");

KeDetachProcess();

}

The DisplayPageDirectory() function accepts the PEB for the process whose page directory is to be printed. The function first calls the KeAttachProcess() function with the given PEB as the parameter. This switches the page directory to the desired one. Still, the function can access the local variables because the kernel address space is shared by all the processes. Now the address space is switched, and the 0xC030000 address points to the page directory to be printed. The function prints the 1024 entries from the page directory and then switches back to the original address space using the KeDetachProcess() function.

void DisplayPageDirectoryForAllProcesses()

{

PLIST_ENTRY ProcessListHead, ProcessListPtr;

ULONG BuildNumber;

ULONG ListEntryOffset;

ULONG NameOffset;

BuildNumber=NtBuildNumber & 0x0000FFFF;

if ((BuildNumber==0x421) || (BuildNumber==0x565)) { // NT

3.51 or NT 4.0

ListEntryOffset=0x98;

NameOffset=0x1DC;

} else if (BuildNumber==0x755) {// Windows 2000 beta2

ListEntryOffset=0xA0;

NameOffset=0x1FC;

} else {

DbgPrint("Unsupported NT Version\n");

return;

}

ProcessListHead=ProcessListPtr=(PLIST_ENTRY)(((char

*)PsInitialSystemProcess)+ListEntryOffset);

while (ProcessListPtr->Flink!=ProcessListHead) {

void *Peb;

char ProcessName[16];

Peb=(void *)(((char *)ProcessListPtr)-

ListEntryOffset);

memset(ProcessName, 0, sizeof(ProcessName));

memcpy(ProcessName, ((char *)Peb)+NameOffset, 16);

DbgPrint("**%s Peb @%x** ", ProcessName, Peb);

DisplayPageDirectory(Peb);

ProcessListPtr=ProcessListPtr->Flink;

}

}

The DisplayPageDirectoryForAllProcesses() function calls the DisplayPageDirectory() function for each process in the system. All the processes running in a system are linked in a list. The function gets hold of the list of the processes from the PEB of the initial system process. The PsInitialSystemProcess variable in NTOSKRNL holds the PEB for the initial system process. The process list node is located at an offset of 0x98 (0xA0 for Windows NT 5.0) inside the PEB. The process list is a circular linked list. Once you get hold of any node in the list, you can traverse the entire list. The DisplayPageDirectoryForAllProcesses() function completes a traversal through the processes list by following the Flink member, printing the page directory for the next PEB in the list every time until it reaches back to the PEB it started with. For every process, the function first prints the process name that is stored at a version-dependent offset within the PEB and then calls the DisplayPageDirectory() function to print the page directory.

Here, we list partial output from the sample program. Please note a couple of things in the following output. First, every page directory has 50-odd valid entries while the page directory size is 1024. The remaining entries are invalid, meaning that the corresponding page tables are either not used or are swapped out. In other words, the main memory overhead of storing page tables is negligible because the page tables themselves can be swapped out. Also, note that the page directories have the same entries in the later portion of the page directory. This is because this part represents the kernel portion shared across all processes by using the same set of page tables for the kernel address range.

Listing 4-12: Displaying page directories: output

**System Peb @fdf06b60**

00500000 008cf000 008ce000 00032000 00034000 00035000 ... ... ...

00040000 00041000 00042000 00043000 00044000 00045000 ... ... ...

00048000 00049000 0004a000 0004b000 0004c000 0004d000 ... ... ...

00050000 00051000 00052000 00053000 00054000 00055000 ... ... ...

00058000 00059000 0005a000 0005b000 0005c000 0005d000 ... ... ...

00020000 00021000 00023000 0040b000 0040c000 0040d000 ... ... ...

00410000 00411000 00412000 00413000 00414000 00415000 ... ... ...

**smss.exe Peb @fe2862e0**

00032000 00034000 00035000 00033000 00e90000 00691000 ... ... ...

00043000 00044000 00045000 00046000 00047000 00048000 ... ... ...

0004b000 0004c000 0004d000 0004e000 0004f000 00050000 ... ... ...

00053000 00054000 00055000 00056000 00057000 00058000 ... ... ...

0005b000 0005c000 0005d000 0005e000 0005f000 00020000 ... ... ...

0040b000 0040c000 0040d000 0040e000 0040f000 00410000 ... ... ...

00413000 00414000 00415000 00416000 00031000

... ... ...

**winlogon.exe Peb @fe27dde0**

00032000 00034000 00035000 00033000 00be1000 00953000 ... ... ...

00043000 00044000 00045000 00046000 00047000 00048000 ... ... ...

0004b000 0004c000 0004d000 0004e000 0004f000 00050000 ... ... ...

00053000 00054000 00055000 00056000 00057000 00058000 ... ... ...

0005b000 0005c000 0005d000 0005e000 0005f000 00020000 ... ... ...

0040b000 0040c000 0040d000 0040e000 0040f000 00410000 ... ... ...

00413000 00414000 00415000 00416000 00031000

... ... ...

DIFFERENCES BETWEEN WINDOWS NT AND WINDOWS 95/98

Generally, the memory management features offered by Windows 95/98 are the same as those in Windows NT. Windows 95/98 also offers 32-bit flat separate address space for each process. Features such as shared memory are still available. However, there are some differences. These differences are due to the fact that Windows 95/98 is not as secure as Windows NT. Many times, Windows 95/98 trades off security for performance reasons. Windows 95/98 still has the concept of user-mode and kernel-mode code. The bottom 3GB is user-mode space, and the top 1GB is kernel-mode space. But the 3GB user-mode space can be further divided into shared space and private space for Windows 95/98. The 2GB to 3GB region is the shared address space for Windows 95/98 processes. For all processes, the page tables for this shared region point to the same set of physical pages.

All the shared DLLs are loaded in the shared region. All the system DLLs–for example, KERNEL32.DLL and USER32.DLL–are shared DLLs. Also, a DLL’s code/data segment can be declared shared while compiling the DLL, and the DLL will get loaded in the shared region. The shared memory blocks are also allocated space in the shared region. In Windows 95/98, once a process maps a shared section, the section is visible to all processes. Because this section is mapped in shared region, other processes need not map it separately.

There are advantages as well as disadvantages of having such a shared region. Windows 95/98 need not map the system DLLs separately for each process; the corresponding entries of page table directory can be simply copied for each process. Also, the system DLLs loaded in shared region can maintain global data about all the processes and separate subsystem processes are not required. Also, most system calls turn out to be simple function calls to the system DLLs, and as a result are very fast. In Windows NT, most system calls either cause a context switch to kernel mode or a context switch to the subsystem process, both of which are costly operations. For developers, loading system DLLs in a shared region means that they can now put global hooks for functions in system DLLs.

For all these advantages, Windows 95/98 pays with security features. In Windows 95/98, any process can access all the shared data even if it has not mapped it. It can also corrupt the system DLLs and affect all processes.

SUMMARY

In this chapter, we discussed the memory management of Windows NT from three different perspectives. Memory management offers programmers a 32-bit flat address space for every process. A process cannot access another process’s memory or tamper with it, but two processes can share memory if they need to. Windows NT builds its memory management on top of the memory management facilities provided by the microprocessor. The 386 (and above) family of Intel microprocessors provides support for segmentation plus paging. The address translation mechanism first calculates the virtual address from the segment descriptor and the specified offset within the segment. The virtual address is then converted to a physical address using the page tables. The operating system can restrict access to certain memory regions by using the security mechanisms that are provided both at the segment level and the page level.

Windows NT memory management provides the programmer with flat address space, data sharing, and so forth by selectively using the memory management features of the microprocessor. The virtual memory manager takes care of the paging and allows 4GB of virtual address space for each process, even when the entire system has much less physical memory at its disposal. The virtual memory manager keeps track of all the physical pages in the system through the page frame database (PFD). The system also keeps track of the virtual address space for each process using the virtual address descriptor (VAD) tree. Windows NT uses the copy-on-write mechanism for various purposes, especially for sharing the DLL data pages. The memory manager has an important part in switching the processor context when a process is scheduled for execution. Windows 95/98 memory management is similar to Windows NT memory management with the differences being due to the fact that Windows 95/98 is not as security conscious as Windows NT.

Memory Management -- Part 5

2007-11-22T11:31:00.002-08:00

Memory Management

IMPACT ON HOOKING

Now we’ll look at the impact of the memory management scheme explained in the last section in the area of hooking DLL API calls. To hook a function from a DLL, you need to change the first few bytes from the function code. As you saw earlier, the DLL code is shared by all processes and is write protected so that a misbehaving process cannot affect other processes. Does this mean that you cannot hook a function in Windows NT? The answer is, “Hooking is possible under Windows NT, but you need to do a bit more work to comply with stability requirements.” Windows NT provides a system call, VirtualProtect, that you can use to change page attributes. Hence, hooking is now a two-step process: Change the attributes of the page containing DLL code to read-write, and then change the code bytes.

Copy-on-Write
“Eureka!” you might say, “I violated Windows NT security. I wrote to a shared page used by other processes also.” No! You did not do that. You changed only your copy of the DLL code. The DLL code page was being shared while you did not write to the page. The moment you wrote on that page, a separate copy of it was made, and the writes went to this copy. All other processes are safely using the original copy of the page. This is how Windows NT protects processes from each other while consuming as few resources as possible.

The VirtualProtect() function does not mark the page as read-write–it keeps the page as read-only. Nevertheless, to distinguish this page from normal read-only pages, it is marked for copy-on-write. Windows NT uses one of the available PTE bits for doing this. When this page is written onto, because it is a read-only page, the processor raises a page fault exception. The page fault handler makes a copy of the page and modifies the page table of the faulting process accordingly. The new copy is marked as read-write so that the process can write to it.

Windows NT itself uses the copy-on-write mechanism for various purposes. The DLL data pages are shared with the copy-on-write mark. Hence, whenever a process writes to a data page, it gets a personal copy of it. Other processes keep sharing the original copy, thus maximizing the sharing and improving memory usage.

A DLL may be loaded in memory at different linear address for different processes. The memory references–for example, address for call instruction, address for a memory to register move instruction, and so on–in the DLL need to be adjusted (patched) depending on the linear address where the DLL gets loaded. This process is called as relocating the DLL. Obviously, relocation has to be done separately for each process. While relocating, Windows NT marks the DLL code pages as copy-on-write temporarily. Thus, only the pages requiring page relocation are copied per process. Other pages that do not have memory references in them are shared by all processes.

This is the reason Microsoft recommends that a DLL be given a preferred base address and be loaded at that address. The binding of the DLL to a specific base address ensures that the DLL need not be relocated if it is loaded at the specified base address. Hence, if all processes load the DLL at the preferred base address, all can share the same copy of DLL code.

The POSIX subsystem of Windows NT uses the copy-on-write mechanism to implement the fork system call. The fork system call creates a new process as a child of a calling process. The child process is a replica of the parent process, and it has the same state of code and data pages as the parent. Since these are two different processes, the data pages should not be shared by them. However, generally it is wasteful to make a copy of the parent’s data pages because in most cases the child immediately invokes the exec system call. The exec system call discards the current memory image of the process, loads a new executable module, and starts executing the new executable module. To avoid copying the data pages, the fork system call marks the data pages as copy-on-write. Hence, a data page is copied only if the parent or the child writes to it.

Copy-on-write is an extremely important concept contributing to the efficiency of NT memory management.

The following sample program demonstrates how copy-on-write works. By running two instances of the program, you can see how the concepts described in this section work. The application loads a DLL, which contains two functions and two data variables. One function does not refer to the outside world, so no relocations are required for it. The other function accesses one global variable, so it contains relocatable instructions or instructions that need relocation. One data variable is put in a shared data section so it will be shared across multiple instances of DLL. One variable is put in a default data section. The two functions are put in separate code sections just to make them page aligned.

When you run the first instance of the application, the application loads and prints the physical addresses of two functions and two data variables. After this, you run the second instance of the same application. In the second instance, the application arranges to load the DLL at a different base address than that of the first instance. Then it prints the physical addresses of two functions and two data variables. Next, the application arranges to load the DLL at the same base address as that of the first instance. In this case, all physical pages are seen to be shared. Next, the application modifies the shared and nonshared variable and modifies the first few bytes of one function, and it prints the physical addresses for two functions and two variables again. We first discuss the code for this sample program and then describe how the output from the sample program demonstrates memory sharing and the effects of the copy-on-write mechanism.

Listing 4-8: SHOWPHYS.C

#include

#include

#include "gate.h"

#include "getphys.h"

HANDLE hFileMapping;

/* Imported function/variable addresses */

static void *NonRelocatableFunction = NULL;

static void *RelocatableFunction = NULL;

static void *SharedVariable = NULL;

static void *NonSharedVariable = NULL;

HINSTANCE hDllInstance;

The initial portion of the file contains the header inclusion and global variable definitions. The program demonstrates the use of various page attributes, especially to implement the copy-on-write mechanism. As described earlier, the program uses four different types of memory sections. The pointers to the four different types of memory sections are defined as global variables. The hDllInstance stores the instance of the instance handle of the DLL that contains the different kind of memory sections used in this demonstration.

/* Loads MYDLL.DLL and initializes addresses of

* imported functions/variables from MYDLL.DLL and

* locks the imported areas

*/

int LoadDllAndInitializeVirtualAddresses()

{

hDllInstance = LoadLibrary("MYDLL.DLL");

if (hDllInstance == NULL) {

printf("Unable to load MYDLL.DLL\n");

return -1;

}

printf("MYDLL.DLL loaded at base address = %x\n",

hDllInstance);

NonRelocatableFunction =

GetProcAddress(GetModuleHandle("MYDLL"),

"_NonRelocatableFunction@0");

RelocatableFunction =

GetProcAddress(GetModuleHandle("MYDLL"),

"_RelocatableFunction@0");

SharedVariable =

GetProcAddress(GetModuleHandle("MYDLL"),

"SharedVariable");

NonSharedVariable =

GetProcAddress(GetModuleHandle("MYDLL"),

"NonSharedVariable");

if((!NonRelocatableFunction) ||

(!RelocatableFunction) ||

(!SharedVariable) ||

(!NonSharedVariable)) {

printf("Unable to get the virtual addresses for"

"imports from MYDLL.DLL\n");

FreeLibrary(hDllInstance);

HDllInstance = 0;

return -1;

}

VirtualLock(NonRelocatableFunction, 1);

VirtualLock(RelocatableFunction, 1);

VirtualLock(SharedVariable, 1);

VirtualLock(NonSharedVariable, 1);

return 0;

}

The four different types of memory sections that we use for the demonstration reside in MYDLL.DLL. The LoadDllAndInitializeVirtualAddresses() function loads MYDLL.DLL in the calling process’s address space and initializes the global variables to point to different types of memory sections in the DLL. The function uses the GetProcAddress() function to get hold of pointers to the exported functions and variables in MYDLL.DLL. The function stores the instance handle for MYDLL.DLL in a global variable so that the FreeDll() function can later use it to unload the DLL. The function also locks the different memory sections so that the pages are loaded in memory and the page table entries are valid. Generally, Windows NT does not load the page table entries unless the virtual address is actually accessed. In other words, the memory won’t be paged in unless accessed. Also, the system can page out the memory that is not used for some time, again marking the page table entries as invalid. We use the VirtualLock() function to ensure that the pages of interest are always loaded and the corresponding page table entries remain valid.

/* Unlocks the imported areas and frees the MYDLL.DLL

*/

void FreeDll()

{

VirtualUnlock(NonRelocatableFunction, 1);

VirtualUnlock(RelocatableFunction, 1);

VirtualUnlock(SharedVariable, 1);

VirtualUnlock(NonSharedVariable, 1);

FreeLibrary(hDllInstance);

HDllInstance = 0;

NonRelocatableFunction = NULL;

RelocatableFunction = NULL;

SharedVariable = NULL;

NonSharedVariable = NULL;

}

The FreeDll() function uses the VirtualUnlock() function to unlock the memory locations locked by the LoadDllAndInitializeVirtualAddresses() function. The function unloads MYDLL.DLL after unlocking the memory locations from the DLL. As the DLL is unloaded, the global pointers to the memory sections in the DLL become invalid. The function sets all these pointers to NULL according to good programming practice.

/* Converts the page attributes in readable form

*/

char *GetPageAttributesString(unsigned int PageAttr)

{

static char buffer[100];

strcpy(buffer, "");

strcat(buffer, (PageAttr&0x01)? "P ": "NP ");

strcat(buffer, (PageAttr&0x02)? "RW ": "R ");

strcat(buffer, (PageAttr&0x04)? "U ": "S ");

strcat(buffer, (PageAttr&0x40)? "D ": " ");

return buffer;

}

The GetPageAttributesString() function returns a string with characters showing the page attributes given the page attribute flags. The LSB in the page attributes indicates whether the page is present in memory or the page table entry is invalid. This information is printed as P or NP, which stands for present or not present. Similarly, R or RW means a read-only or read-write page; S or U means a supervisor-mode or a user-mode page; and D means a dirty page. The various page attributes are represented by different bits in the PageAttr parameter to this function. The function checks the bits and determines whether the page possesses the particular attributes.

/* Displays virtual to physical address mapping

*/

int DisplayVirtualAndPhysicalAddresses()

{

DWORD pNonRelocatableFunction = 0;

DWORD pRelocatableFunction = 0;

DWORD pSharedVariable = 0;

DWORD pNonSharedVariable = 0;

DWORD aNonRelocatableFunction = 0;

DWORD aRelocatableFunction = 0;

DWORD aSharedVariable = 0;

DWORD aNonSharedVariable = 0;

printf("\nVirtual to Physical address mapping\n");

printf("\n------------------------------------\n");

printf("Variable/function Virtual Physical Page\n");

printf(" Address Address Attributes\n");

printf("--------------------------------------\n");

GetPhysicalAddressAndPageAttributes(

NonRelocatableFunction,

&pNonRelocatableFunction, &aNonRelocatableFunction);

GetPhysicalAddressAndPageAttributes(

RelocatableFunction,

&pRelocatableFunction, &aRelocatableFunction);

GetPhysicalAddressAndPageAttributes(

SharedVariable,

&pSharedVariable,

&aSharedVariable);

GetPhysicalAddressAndPageAttributes(

NonSharedVariable,

&pNonSharedVariable,

&aNonSharedVariable);

printf("NonRelocatableFunction\t %8x\t %8x\t %s\n",

NonRelocatableFunction,

pNonRelocatableFunction,

GetPageAttributesString(

aNonRelocatableFunction));

printf("RelocatableFunction\t %8x\t %8x\t %s\n",

RelocatableFunction,

pRelocatableFunction,

GetPageAttributesString(

aRelocatableFunction));

printf("SharedVariable\t %8x\t %8x\t %s\n",

SharedVariable,

pSharedVariable,

GetPageAttributesString(

aSharedVariable));

printf("NonSharedVariable\t %8x\t %8x\t %s\n",

NonSharedVariable,

pNonSharedVariable,

GetPageAttributesString(

aNonSharedVariable));

printf("------------------------------------\n\n");

return 0;

}

The DisplayVirtualAndPhysicalAddresses() function is a utility function that displays the virtual address, the physical address, and the page attributes for different memory sections. It uses the global pointers to the different sections in MYDLL.DLL initialized by the LoadDllAndInitializeVirtualAddresses() function. It uses the GetPhysicalAddressAndPageAttributes() function to get hold of the physical page address and the page attributes for the given virtual address. The first parameter to the GetPhysicalAddressAndPageAttributes() function is the input virtual address. The function fills in the physical address for the input virtual address in the memory location pointed to by the second parameter and the page attributes in the location pointed to by the third parameter.

int FirstInstance()

{

printf("***This is the first instance of the"

" showphys program***\n\n");

printf("Loading DLL MYDLL.DLL\n");

if (LoadDllAndInitializeVirtualAddresses()!=0) {

return -1;

}

DisplayVirtualAndPhysicalAddresses();

printf("Now Run another copy of showphys ...\n");

getchar();

FreeDll();

}

Memory Management -- Part 5

2007-11-22T11:31:00.001-08:00

Memory Management

The first two members dictate the address range represented by the VAD node. Each VAD tree node maintains a pointer to the parent node and a pointer to the left child and the right child. The VAD tree is a binary tree. For every node in the tree, the left subtree consists of nodes representing lower address ranges, and the right subtree consists of nodes representing the higher address ranges. The last member in the VAD node is the flags for the address range.

The VADDUMP.C file has a few other global variables apart from the VadInfoArray. A couple of global variables are used while locating the root of the VAD tree. The PEB of a process points to the VAD tree root for that process. The offset of this pointer inside the PEB varies with the Windows NT version. We set the VadRootOffset to the appropriate offset value of the VAD root pointer depending on the Windows NT version. There is a similar problem of Windows NT version dependency while accessing the PEB for the process. We use the Thread Environment Block (TEB) to get to the PEB. One field in TEB points to the PEB, but the offset of this field inside the TEB structure varies with the Windows NT version. We set the PebOffset variable to the appropriate offset value of the PEB pointer inside the TEB structure depending on the Windows NT version. Another global variable, NtVersion, stores the version of Windows NT running on the machine.

That leaves us with two more global variables, namely, VadInfoArrayIndex and VadTreeRoot. The VadInfoArrayIndex is the number of initialized entries in the VadInfoArray. The VadInfoArray entries after VadInfoArrayIndex are free. The VadTreeRoot variable stores the root of the VAD tree.

The sample has been tested on Windows NT 3.51, 4.0 and Windows 2000 beta2. The sample will run on other versions of Windows 2000, provided the offsets of VadRoot and PEB remain same.

/* Recursive function which walks the vad tree and

* fills up the global VadInfoArray with the Vad

* entries. Function is limited by the

* MAX_VAD_ENTRIES. Other VADs after this are not

* stored

*/

void _stdcall VadTreeWalk(PVAD VadNode)

{

if (VadNode == NULL) {

return;

}

if (VadInfoArrayIndex >= MAX_VAD_ENTRIES) {

return;

}

VadTreeWalk(VadNode->LeftLink);

VadInfoArray[VadInfoArrayIndex].VadLocation = VadNode;

VadInfoArray[VadInfoArrayIndex].Vad.StartingAddress =

VadNode->StartingAddress;

VadInfoArray[VadInfoArrayIndex].Vad.EndingAddress =

VadNode->EndingAddress;

if (NtVersion == 5) {

(DWORD)VadInfoArray[VadInfoArrayIndex].

Vad.StartingAddress <<= 12;

(DWORD)VadInfoArray[VadInfoArrayIndex].

Vad.EndingAddress += 1;

(DWORD)VadInfoArray[VadInfoArrayIndex].

Vad.EndingAddress <<= 12;

(DWORD)VadInfoArray[VadInfoArrayIndex].

Vad.EndingAddress -= 1;

}

VadInfoArray[VadInfoArrayIndex].Vad.ParentLink =

VadNode->ParentLink;

VadInfoArray[VadInfoArrayIndex].Vad.LeftLink =

VadNode->LeftLink;

VadInfoArray[VadInfoArrayIndex].Vad.RightLink =

VadNode->RightLink;

VadInfoArray[VadInfoArrayIndex].Vad.Flags =

VadNode->Flags;

VadInfoArrayIndex++;

VadTreeWalk(VadNode->RightLink);

}

The VadTreeWalk() function is executed in the kernel mode using the callgate mechanism. The function traverses the VAD tree in the in-order fashion and fills up the VadInfoArray. The function simply returns if the node pointer parameter is NULL or the VadInfoArray is full. Otherwise, the function recursively calls itself for the left subtree. The recursion is terminated when the left child pointer is NULL. The function then fills up the next free entry in the VadInfoArray and increments the VadInfoArrayIndex to point to the next free entry. Windows 2000 stores the page numbers instead of the actual addresses in VAD. Hence, for Windows 2000, we need to calculate the starting address and the ending address from the page numbers stored in these fields. As the last step in the in-order traversal, the function issues a self-recursive to process the right subtree.

/* C function called through assembly stub */

void _stdcall CFuncDumpVad(PVAD VadRoot)

{

VadTreeRoot = VadRoot;

VadInfoArrayIndex = 0;

VadTreeWalk(VadRoot);

}

The CfuncDumpVad is the caller of the VadTreeWalk() function. It just initializes the global variables used by the VadTreeWalk() function and calls the VadTreeWalk() function for the root of the VAD tree.

/* Displays the Vad tree */

void VadTreeDisplay()

{

int i;

printf("VadRoot is located @%08x\n\n",

VadTreeRoot);

printf("Vad@\t Starting\t Ending\t Parent\t "

"LeftLink\t RightLink\n");

for (i=0; i < VadInfoArrayIndex; i++) {

printf("%08x %08x %08x %8x %08x %08x\n",

VadInfoArray[i].VadLocation,

VadInfoArray[i].Vad.StartingAddress,

VadInfoArray[i].Vad.EndingAddress,

VadInfoArray[i].Vad.ParentLink,

VadInfoArray[i].Vad.LeftLink,

VadInfoArray[i].Vad.RightLink);

}

printf("\n\n");

}

The VadTreeDisplay() function is a very simple function that is executed in user mode. The function iterates through all the entries initialized by the VadTreeWalk() function and prints the entries. Essentially, the function prints the VAD tree in the infix order because the VadTreeWalk() function dumps the VAD tree in the infix order.

void SetDataStructureOffsets()

{

switch (NtVersion) {

case 3:

PebOffset = 0x40;

VadRootOffset = 0x170;

break;

case 4:

PebOffset = 0x44;

VadRootOffset = 0x170;

break;

case 5:

PebOffset = 0x44;

VadRootOffset = 0x194;

break;

}

}

As we described earlier, the offset of the PEB pointer within TEB and the offset of the VAD root pointer within the PEB are dependent on the Windows NT version. The SetDataStructureOffsets() function sets the global variables indicating these offsets depending on the Windows NT version.

main()

{

WORD CallGateSelector;

int rc;

short farcall[3];

void DumpVad(void);

void *ptr;

OSVERSIONINFO VersionInfo;

VersionInfo.dwOSVersionInfoSize = sizeof(VersionInfo);

if (GetVersionEx(&VersionInfo) == TRUE) {

NtVersion = VersionInfo.dwMajorVersion;

}

if ((NtVersion < 3)||(NtVersion > 5)) {

printf("Unsupported NT version, exiting...");

return 0;

}

SetDataStructureOffsets();

/* Creates call gate to read vad tree from Ring 3

*/

rc = CreateCallGate(DumpVad, 0, &CallGateSelector);

if (rc != SUCCESS) {

printf("CreateCallGate failed, rc=%x\n", rc);

return 1;

}

farcall[2] = CallGateSelector;

_asm {

call fword ptr [farcall]

}

printf("Dumping the Vad tree ...\n\n");

VadTreeDisplay();

printf("Allocating memory using VirtualAlloc");

ptr = VirtualAlloc(NULL, 4096, MEM_COMMIT,

PAGE_READONLY);

if (ptr == NULL) {

printf("Unable to allocate memory\n");

goto Quit;

}

printf("\nMemory allocated @%x\n", ptr);

_asm {

call fword ptr [farcall]

}

printf("\n\nDumping the Vad tree again...\n\n");

VadTreeDisplay();

Quit:

rc = FreeCallGate(CallGateSelector);

if (rc != SUCCESS) {

printf("FreeCallGate failed, Selector=%x, rc=%x\n",

CallGateSelector, rc);

}

return 0;

}

The main() function starts by getting the Windows NT version and calling SetDataStructureOffsets() to set the global variables storing the offsets for the PEB and the VAD tree root. It then creates a callgate in the same manner as in the SHOWDIR sample program. Issuing a call through this callgate ultimately results in the execution of the VadTreeWalk() function that fills up the VadInfoArray. The main() function then calls the VadTreeDisplay() function to print the VadInfoArray entries.

We also show you the change in the VAD tree due to memory allocation in this sample program. After printing the VAD tree once, the program allocates a chunk of memory. Then, the program issues the callgate call again and prints the VAD tree after returning from the call. You can observe the updates that happened to the VAD tree because of the memory allocation. The program frees up the callgate before exiting.

Listing 4-6: RING0.ASM

.386

.model small

.code

public _DumpVad

extrn _CFuncDumpVad@4:near

extrn _PebOffset:near

extrn _VadRootOffset:near

include ..\include\undocnt.inc

_DumpVad proc

Ring0Prolog

;Gets the current thread

MOV EAX,FS:[00000124h]

;Gets the current process

ADD EAX, DWORD PTR [_PebOffset]

MOV EAX,[EAX]

;Push Vad Tree root

ADD EAX, DWORD PTR [_VadRootOffset]

MOV EAX, [EAX]

PUSH EAX

CALL _CFuncDumpVad@4

Ring0Epilog

RETF

_DumpVad endp

END

The function to be called from the callgate needs to be written in the Assembly language for reasons already described. The DumpVad() function gets hold of the VAD root pointer and calls the CFuncDumpVad() function that dumps the VAD tree in the VadInfoArray. The function gets hold of the VAD root from the PEB after getting hold of the PEB from the TEB. The TEB of the currently executing thread is always pointed to by FS:128h. As described earlier, the offset of the VAD root pointer inside PEB and the offset of the PEB pointer inside the TEB vary with the Windows NT version. The DumpVad() function uses the offset values stored in the global variable by the SetDataStructureOffsets() function.

Listing 4-7 presents the output from an invocation of the VADDUMP program. Note that the VAD tree printed after allocating memory at address 0x300000 shows an additional entry for that address range.

Listing 4-7: Program output

Dumping the Vad tree...

VadRoot is located @fe21a9c8

Vad@ Starting Ending Parent LeftLink RightLink

fe216b08 00010000 00010fff fe21a9c8 00000000 fe25a0e8

fe25a0e8 00020000 00020fff fe216b08 00000000 fe275da8

fe275da8 00030000 0012ffff fe25a0e8 00000000 fe22a428

fe22a428 00130000 00130fff fe275da8 00000000 fe26b328

fe26b328 00140000 0023ffff fe22a428 00000000 fe210fc8

fe210fc8 00240000 0024ffff fe26b328 00000000 fe21a8c8

fe21a8c8 00250000 00258fff fe210fc8 00000000 fe21be68

fe21be68 00260000 0026dfff fe21a8c8 00000000 fe215dc8

fe215dc8 00270000 002b0fff fe21be68 00000000 fe231e88

fe231e88 002c0000 002c0fff fe215dc8 00000000 fe2449e8

fe2449e8 002d0000 002dffff fe231e88 00000000 fe21cb48

fe21cb48 002e0000 002e0fff fe2449e8 00000000 fe23b7a8

fe23b7a8 002f0000 002fffff fe21cb48 00000000 00000000

fe21a9c8 00400000 0040cfff 0 fe216b08 fe23c488

fe21b3e8 10000000 1000dfff fe2333e8 00000000 fe226348

fe2176c8 77e20000 77e4bfff fe226348 00000000 fe2326e8

fe2152c8 77e50000 77e54fff fe2326e8 00000000 00000000

fe2326e8 77e60000 77e9bfff fe2176c8 fe2152c8 00000000

fe226348 77ea0000 77ed7fff fe21b3e8 fe2176c8 fe2197c8

fe2197c8 77ee0000 77f12fff fe226348 00000000 00000000

fe2333e8 77f20000 77f73fff fe23c488 fe21b3e8 00000000

fe23c488 77f80000 77fcdfff fe21a9c8 fe2333e8 fe25aa88

fe22b408 7f2d0000 7f5cffff fe25aa88 00000000 fe22c4a8

fe22c4a8 7f5f0000 7f7effff fe22b408 00000000 fe23f5e8

fe23f5e8 7ff70000 7ffaffff fe22c4a8 00000000 00000000

fe25aa88 7ffb0000 7ffd3fff fe23c488 fe22b408 fe218288

fe21da88 7ffde000 7ffdefff fe218288 00000000 00000000

fe218288 7ffdf000 7ffdffff fe25aa88 fe21da88 00000000

Allocating memory using VirtualAlloc

Memory allocated @300000

Dumping the Vad tree again...

VadRoot is located @fe21a9c8

Vad@ Starting Ending Parent LeftLink RightLink

fe216b08 00010000 00010fff fe21a9c8 00000000 fe25a0e8

fe25a0e8 00020000 00020fff fe216b08 00000000 fe275da8

fe275da8 00030000 0012ffff fe25a0e8 00000000 fe22a428

fe22a428 00130000 00130fff fe275da8 00000000 fe26b328

fe26b328 00140000 0023ffff fe22a428 00000000 fe210fc8

fe210fc8 00240000 0024ffff fe26b328 00000000 fe21a8c8

fe21a8c8 00250000 00258fff fe210fc8 00000000 fe21be68

fe21be68 00260000 0026dfff fe21a8c8 00000000 fe215dc8

fe215dc8 00270000 002b0fff fe21be68 00000000 fe231e88

fe231e88 002c0000 002c0fff fe215dc8 00000000 fe2449e8

fe2449e8 002d0000 002dffff fe231e88 00000000 fe21cb48

fe21cb48 002e0000 002e0fff fe2449e8 00000000 fe23b7a8

fe23b7a8 002f0000 002fffff fe21cb48 00000000 fe27b628

fe27b628 00300000 00300fff fe23b7a8 00000000 00000000

fe21a9c8 00400000 0040cfff 0 fe216b08 fe23c488

fe21b3e8 10000000 1000dfff fe2333e8 00000000 fe226348

fe2176c8 77e20000 77e4bfff fe226348 00000000 fe2326e8

fe2152c8 77e50000 77e54fff fe2326e8 00000000 00000000

fe2326e8 77e60000 77e9bfff fe2176c8 fe2152c8 00000000

fe226348 77ea0000 77ed7fff fe21b3e8 fe2176c8 fe2197c8

fe2197c8 77ee0000 77f12fff fe226348 00000000 00000000

fe2333e8 77f20000 77f73fff fe23c488 fe21b3e8 00000000

fe23c488 77f80000 77fcdfff fe21a9c8 fe2333e8 fe25aa88

fe22b408 7f2d0000 7f5cffff fe25aa88 00000000 fe22c4a8

fe22c4a8 7f5f0000 7f7effff fe22b408 00000000 fe23f5e8

fe23f5e8 7ff70000 7ffaffff fe22c4a8 00000000 00000000

fe25aa88 7ffb0000 7ffd3fff fe23c488 fe22b408 fe218288

fe21da88 7ffde000 7ffdefff fe218288 00000000 00000000

fe218288 7ffdf000 7ffdffff fe25aa88 fe21da88 00000000

The output of the VADDUMP program does not really look like a tree. You have to trace through the output to get the tree structure. The entry with a null parent link is the root of the tree. Once you find the root, you can follow the child pointers. To follow a child pointer, search the pointer in the first column, named Vad@, in the output. The Vad entry with the same Vad@ is the entry for the child that you are looking for. An all-zero entry for a left/right child pointer indicates that there is no left/right subtree for the node. Figure 4-5 shows a partial tree constructed from the output shown previously.

Memory Management -- Part 3

2007-11-22T11:29:00.000-08:00


Memory Management

Let’s analyze, one step at a time, why the two entries are different. The page tables themselves need to be mapped onto some linear address. When Windows NT needs to access the page tables, it uses this linear address range. To represent 4GB of memory divided into 1MB pages of 4K each, we need 1K page tables each having 1K entries. To map these 1K page tables, Windows NT reserves 4MB of linear address space in each process. As we saw earlier, each process has a different set of page tables. Whatever the process, Windows NT maps the page tables on the linear address range from 0xC0000000 to 0xC03FFFFF. Let’s call this linear address range as the page table address range. In other words, the page table address range maps to different page tables–that is, to different physical pages–for different processes. As you may have noticed, the page table addresses range falls in the kernel address space. Windows NT cannot map this crucial system data structure in the user address space and allow user-mode processes to play with the memory. Ultimately, the result is that two processes cannot share pages in the page table address range although the addresses lie in the kernel-mode address range.

Exactly one page table is required to map 4MB address space because each page table has 1K entries and each entry corresponds to a 4K page. Consequently, Windows NT cannot share the page table corresponding to the page table address range. This accounts for one of the two mysterious entries in the page table directory. However, the entry’s mystery does not end here–there is one more subtle twist to this story. The physical address specified in this entry matches the physical address of the page table directory. The obvious conclusion is that the page table directory acts also as the page table for the page table address range. This is  possible because the formats of the page table directory entry and PTE are the same on 80386.

The processor carries out an interesting sequence of actions when the linear address within the page table address range is translated to a physical address. Let’s say that the CR3 register points to page X. As the first step in the address translation process, the processor treats the page X as the page table directory and finds out the page table for the given linear address. The page table happens to be page X again. The processor now treats page X as the required page table and finds out the physical address from it. A more interesting case occurs when the operating system is accessing the page table directory itself. In this case, the physical address also falls in page X!

Let’s now turn to the second mysterious entry. The 4MB area covered by this page directory entry is internally referred to as hyperspace. This area is used for mapping the physical pages belonging to other processes into virtual address space. For example, a function such as MmMapPageInHyperspace() uses the virtual addresses in this range. This area is also used during the early stages of process creation. For example, when a parent process such as PROGMAN.EXE spawns a child process such as NOTEPAD.EXE, PROGMAN.EXE has to create the address space for NOTEPAD.EXE. This is done as a part of the MmCreateProcessAddressSpace() function. For starting any process, an address space must be created for the process. Address space is nothing but page directory. Also, the upper-half entries of page directory are common for all processes except for the two entries that we have already discussed. These entries need to be created for the process being spawned. The MmCreateProcessAddressSpace() function allocates three pages of memory: the first page for the page directory, the second page for holding the hyperspace page table entries, and the third page for holding the working set information for the process being spawned.

Once these pages are allocated, the function maps the first physical page in the address space using the MmMapPageInHyperSpace() function. Note that the MmMapPageInHyperSpace() function runs in the context of PROGMAN.EXE. Now the function copies the page directory entries in the upper half of the page directory to the mapped hyperspace virtual address. In short, PROGMAN.EXE creates the page directory for the NOTEPAD.EXE.

Windows NT supports memory-mapped files. When two processes map the same file, they share the same set of physical pages. Hence, memory-mapped files can be used for sharing memory. In fact, Windows NT itself uses memory-mapped files to load DLLs and executables. If two processes map the same DLL, they automatically share the DLL pages. The memory-mapped files are implemented using the section object under Windows NT. A data structure called PROTOPTE is associated with each section object. This data structure is a variable-length structure based on the size of the section. This data structure contains a 4-byte entry for each page in the virtual address space mapped by the section object. Each 4-byte entry has the same structure as that of the PTE. When the page is not being used by any of the processes, the protopte entry is invalid and contains enough information to get the page back. In this case, the CPU PTE contains a fixed value that is 0xFFFFF480, which indicates that accessing this page will be considered a protopte fault.

Now comes the toughest of all questions: "How can Windows NT give away 4GB of memory to each process when there is far less physical RAM available on the board?" Windows NT, as well as all other operating systems that allow more address space than actual physical memory, uses a technique called virtual memory to achieve this. In the next section, we discuss virtual memory management in Windows NT.


VIRTUAL MEMORY MANAGEMENT

The basic idea behind virtual memory is very simple. For each process, the operating system maps few addresses to real physical memory because RAM is expensive and relatively rare. Remaining memory for each process is really maintained on secondary storage (usually a hard disk). That’s why it is called virtual memory. The addresses that are not mapped on physical RAM are marked as such. Whenever a process accesses such an address, the operating system brings the data into memory from secondary storage. If the operating system runs out of physical RAM, some data is thrown out to make space. We can always get back this data because a copy is maintained on secondary storage. The data to be thrown out is decided by the replacement policy. Windows NT uses First-In-First-Out (FIFO) replacement policy. According to this policy, the oldest data (that is, the data that was brought in the RAM first) is thrown out whenever there is a space crunch.

To implement virtual memory management, Windows NT needs to maintain a lot of data. First, it needs to maintain whether each address is mapped to physical RAM or the data is to be brought in from secondary storage when a request with the address comes. Maintaining this information for each byte itself takes a lot of space (actually, more space than the address space for which the information is to be maintained). So Windows NT breaks the address space into 4KB pages and maintains this information in page tables. As we saw earlier, a page table entry (PTE) consists of the address of the physical page (if the page is mapped to physical RAM) and attributes of the page. Since the processor heavily depends on PTEs for address translation, the structure of PTE is processor dependent.

If a page is not mapped onto physical RAM, Windows NT marks the page as invalid. Any access to this page causes a page fault, and the page fault handler can bring in the page from the secondary storage. To be more specific, when the page contains DLL code or executable module code, the page is brought in from the DLL or executable file. When the page contains data, it is brought in from the swap file. When the page represents a memory-mapped file area, it is brought in from the corresponding file. Windows NT needs to keep track of free physical RAM so that it can allocate space for a page brought in from secondary storage in case of a page fault. This information is maintained in a kernel data structure called the Page Frame Database (PFD). The PFD also maintains a FIFO list of in-memory pages so that it can decide on pages to throw out in case of a space crunch.

Before throwing out a page, Windows NT must ensure that the page is not dirty. Otherwise, it needs to write that page to secondary storage before throwing it out. If the page is not shared, the PFD contains the pointer to PTE so that if the operating system decides to throw out a particular page, it can then go back and mark the PTE as invalid. If the page is shared, the PFD contains a pointer to the corresponding PROTOPTE entry. In this case, the PFD also contains a reference count for the page. A page can be thrown out only if its reference count is 0. In general, the PFD maintains the status of every physical page.

The PFD is an array of 24-byte entries, one for each physical page. Hence, the size of this array is equal to the number of physical pages that are stored in a kernel variable, namely, MmNumberOfPhysicalPages. The pointer to this array is stored in a kernel variable, namely, MmpfnDatabase. A physical page can be in several states–for example, it can be in-use, free, free but dirty, and so on. A PFD entry is linked in a doubly linked list, depending on the state of the physical page represented by it. For example, the PFD entry representing a free page is linked in the free pages list. Figure 4-4 shows these lists linked through the PFD. The forward links are shown on the left side of the PFD, and the backward links are shown on the right side.

There are in all six kinds of lists. The heads of these lists are stored in following kernel variables:

MmStandbyPageListHead

MmModifiedNoWritePageListHead

MmModifiedPageListHead

MmFreePageListHead

MmBadPageListHead

MmZeroedPageListHead

All these list heads are actually structures of 16 bytes each. Here is the structure definition:

typedef struct PageListHead {

 DWORD NumberOfPagesInList,

 DWORD TypeOfList,

 DWORD FirstPage,

 DWORD LastPage

} PageListHead_t;

The FirstPage field can be used as an index into the PFD. The PFD entry contains a pointer to the next page. Using this, you can traverse any of the lists. Here is the structure definition for the PFD entry:

typedef struct PfdEntry {

 DWORD NextPage,

 void *PteEntry/*PpteEntry,

 DWORD PrevPage,

 DWORD PteReferenceCount,

 void *OriginalPte,

 DWORD Flags;

} PfdEntry_t;

Using this, you can easily write a program to dump the PFD. However, there is one problem: kernel variables, such as list heads, MmPfnDatabase, and MmNumberOfPhysicalPages, are not exported. Therefore, you have to deal with absolute addresses, which makes the program dependent on the Windows NT version and build type.


VIRTUAL ADDRESS DESCRIPTORS

Along with the free physical pages, Windows NT also needs to keep track of the virtual address space allocation for each process. Whenever a process allocates a memory block–for example, to load a DLL–Windows NT checks for a free block in the virtual address space, allocates virtual address space, and updates the virtual address map accordingly. The most obvious place to maintain this information is page tables. For each process, Windows NT maintains separate page tables. There are 1 million pages, and each page table entry is 4 bytes. Hence, full page tables for a single process would take 4MB of RAM! There is a solution to this: Page tables themselves can be swapped out. It is inefficient to swap in entire page tables when a process wants to allocate memory. Hence, Windows NT maintains a separate binary search tree containing the information about current virtual space allocation for each process. A node in this binary search tree is called a Virtual Address Descriptor (VAD). For each block of memory allocated to a process, Windows NT adds a VAD entry to the binary search tree. Each VAD entry contains the allocated address range–that is, the start address and the end address of the allocated block, pointers to left and right children VADs, and a pointer to the parent VAD. The process environment block (PEB) contains a pointer, namely, VadRoot, to the root of this tree.

Listing 4-5: VADDUMP.C

/* Should be compiled in release mode */

#define _X86_




#include 

#include 

#include 




#include "undocnt.h"

#include "gate.h"




/*Define the WIN32 calls we are using, since we can not include both

NTDDK.H and

WINDOWS.H in the same ’C’ file.*/

typedef struct _OSVERSIONINFO{

 ULONG dwOSVersionInfoSize;

 ULONG dwMajorVersion;

 ULONG dwMinorVersion;

 ULONG dwBuildNumber;

 ULONG dwPlatformId;

 CCHAR szCSDVersion[ 128 ];

} OSVERSIONINFO, *LPOSVERSIONINFO;

BOOLEAN _stdcall GetVersionExA(LPOSVERSIONINFO);

PVOID _stdcall VirtualAlloc(PVOID, ULONG, ULONG, ULONG);




/* Max vad entries */

#define MAX_VAD_ENTRIES   0x200




/* Following variables are accessed in RING0.ASM */

ULONG NtVersion;

ULONG PebOffset;

ULONG VadRootOffset;




#pragma pack(1)

typedef struct VadInfo {

 void *VadLocation;

 VAD Vad;

} VADINFO, *PVADINFO;

#pragma pack()




VADINFO VadInfoArray[MAX_VAD_ENTRIES];

int VadInfoArrayIndex;

PVAD VadTreeRoot;

The initial portion of the VADDUMP.C file has a few definitions apart from the header inclusion. In this program, we use the callgate mechanism as we did in the showdir program–hence the inclusion of the GATE.H header file. After the header inclusion, the file defines the maximum number of VAD entries that we’ll process. There is no limit on the nodes in a VAD tree. We use the callgate mechanism for kernel-mode execution of a function that dumps the VAD tree in an array accessible from the user mode. This array can hold up to MAX_VAD_ENTRIES entries. Each entry in the array is of type VADINFO. The VADINFO structure has two members: the address of the VAD tree node and the actual VAD tree node. The VAD tree node structure is defined in the UNDOCNT.H file as follows:

typedef struct vad {

void *StartingAddress;

void *EndingAddress;

struct vad *ParentLink;

struct vad *LeftLink;

struct vad *RightLink;

DWORD Flags;

}VAD, *PVAD;