potluri's Code Blog

2023-01-31T16:02:00.000-08:00

SPARQL Property Paths: Advanced Graph Traversal Made Easy

Master SPARQL property paths to efficiently traverse and query complex graph relationships.

Code copied!

🎯 What are Property Paths?

Property paths in SPARQL allow you to express complex graph traversal patterns concisely. They're particularly useful for finding connections between resources that may be several steps apart.

🔧 Basic Path Operators

Operator	Description	Example Use Case
/	Sequence Path	Finding grandparents (parent of parent)
\|	Alternative Path	Finding either authors or scientists
?	Zero or One Path	Optional relationships
*	Zero or More Path	All ancestors in a family tree
+	One or More Path	All descendants in a family tree
!	Negated Property Path	Finding relationships that are not of a specific type

📝 Basic Examples

Finding Grandparents

Sequence Path Example

SELECT ?person ?grandparent
WHERE {
    ?person dbo:parent/dbo:parent ?grandparent .
}

The / operator connects two properties in sequence, similar to following two separate relationships.

Finding Authors or Scientists

Alternative Path Example

SELECT ?person ?name
WHERE {
    ?person rdf:type/(dbo:Author|dbo:Scientist) ;
           foaf:name ?name .
}

🌟 Advanced Path Patterns

Academic Genealogy

One or More Path Example

SELECT ?student ?advisor ?generation
WHERE {
    ?student dbo:doctoralAdvisor+ ?advisor .
    BIND(LENGTH(STR(?advisor)) AS ?generation)
}

The + operator finds one or more repetitions of the relationship, perfect for tracing academic lineage.

Finding Connected Artists

Complex Path Example

SELECT ?artist1 ?artist2 ?path
WHERE {
    ?artist1 rdf:type dbo:Artist .
    ?artist2 rdf:type dbo:Artist .
    ?artist1 (dbo:collaborator|dbo:influenced)* ?intermediate .
    ?intermediate (dbo:collaborator|dbo:influenced)* ?artist2 .
    BIND(CONCAT(?artist1, " -> ", ?intermediate, " -> ", ?artist2) AS ?path)
    FILTER(?artist1 != ?artist2)
}

💡 Best Practices

Follow these guidelines for effective property paths:

Use property paths for traversing known relationship patterns
Be careful with * and + on large datasets
Consider using LIMIT to restrict results when using recursive paths
Use named graphs to limit the scope of path traversal
Combine property paths with FILTER for more precise results

⚠️ Common Pitfalls

Watch out for:

Unbounded path expressions (* and +) on large datasets
Complex path patterns that may timeout
Forgetting to filter out self-references in cyclic paths
Not considering the direction of relationships

🎯 Use Cases

Social network analysis (finding connections between people)
Genealogical research (family trees)
Citation networks (academic papers and their references)
Organizational hierarchies
Knowledge graph navigation

🚀 Advanced Techniques

Limited Length Paths

Path Length Restriction Example

SELECT DISTINCT ?author1 ?author2
WHERE {
    ?author1 rdf:type dbo:Author .
    ?author2 rdf:type dbo:Author .
    ?author1 (dbo:coAuthor){1,3} ?author2 .
    FILTER(?author1 != ?author2)
}

The {1,3} syntax restricts the path length to between 1 and 3 steps.

2022-12-31T15:59:00.000-08:00

Mastering SPARQL: A Comprehensive Guide to Querying RDF Data

Learn how to effectively query RDF data using SPARQL, from basic queries to advanced federated searches.

Code copied!

📚 Introduction to SPARQL

SPARQL (SPARQL Protocol and RDF Query Language) is the standard query language for RDF (Resource Description Framework) data. It allows you to query and manipulate data stored in RDF format across different data sources.

🔍 Basic Queries

Let's start with some basic SPARQL queries:

Finding Capital Cities in Europe

Basic SELECT Query

PREFIX rdf: 
PREFIX dbo: 

SELECT ?city ?country ?population
WHERE {
    ?city rdf:type dbo:City ;
          dbo:country ?country ;
          dbo:populationTotal ?population .
    ?country dbo:capital ?city ;
             dbo:continent dbr:Europe .
}
ORDER BY DESC(?population)
LIMIT 10

Key Points:

PREFIX declarations define shortcuts for URIs
SELECT specifies which variables to include in results
WHERE clause defines the pattern to match
ORDER BY and LIMIT control result ordering and quantity

🔄 Advanced Queries

Using OPTIONAL and FILTER

Advanced Query Example

SELECT ?scientist ?discovery ?year
WHERE {
    ?scientist rdf:type dbo:Scientist .
    OPTIONAL {
        ?discovery dbo:discoverer ?scientist ;
                  dbo:discoveryYear ?year .
    }
    FILTER(?year > 1900)
}

Pro Tip: Use OPTIONAL when some data might be missing and you still want results for the main pattern.

✏️ Update Operations

Inserting and Modifying Data

Update Query Example

PREFIX ex: 

INSERT DATA {
    ex:JohnDoe rdf:type ex:Person ;
               ex:name "John Doe" ;
               ex:age 30 .
}

DELETE {
    ex:JohnDoe ex:age ?oldAge .
}
INSERT {
    ex:JohnDoe ex:age 31 .
}
WHERE {
    ex:JohnDoe ex:age ?oldAge .
}

🌐 Federated Queries

Federated queries allow you to query multiple SPARQL endpoints in a single query:

Federated Query Example

SELECT ?person ?dbpediaBirth ?wikidataBirth
WHERE {
    # DBpedia part
    ?person rdf:type dbo:Scientist ;
            dbo:birthDate ?dbpediaBirth .
    
    # Wikidata part
    SERVICE  {
        ?wikidataPerson wdt:P31 wd:Q5 ;
                        wdt:P569 ?wikidataBirth .
    }
}

📊 Common SPARQL Patterns

Pattern	Use Case
FILTER	Restrict results based on conditions
OPTIONAL	Include optional patterns without failing the query
UNION	Combine results from alternative patterns
GROUP BY	Aggregate results based on variables
SERVICE	Query external SPARQL endpoints

🎯 Best Practices

Follow these guidelines for better queries:

Always use appropriate PREFIX declarations
Start with specific patterns and add OPTIONAL patterns later
Use LIMIT and OFFSET for pagination
Include FILTER clauses as early as possible
Use named graphs for better data organization

graphql

2020-12-31T15:55:00.000-08:00

Building a Scalable GraphQL API with Apollo Server and DataLoader

Learn how to build an efficient GraphQL API using Apollo Server and DataLoader. We'll create a blog-like system with users, posts, and comments while avoiding the N+1 query problem.

Code copied!

🎯 What We're Building

We'll create a GraphQL API that handles relationships between:

Users who write posts and comments
Posts that belong to users and have comments
Comments that belong to posts and users

User ──┬── Posts
       └── Comments ── Posts

🚀 Getting Started

First, let's set up our project with the necessary dependencies:

package.json

{
  "name": "graphql-apollo-tutorial",
  "version": "1.0.0",
  "type": "module",
  "dependencies": {
    "@apollo/server": "^4.9.5",
    "dataloader": "^2.2.2",
    "graphql": "^16.8.1"
  }
}

📦 Setting Up DataLoaders

DataLoader is a key tool for solving the N+1 query problem in GraphQL. It batches and caches database lookups:

What is the N+1 Problem?
When fetching a list of items and their relations, you might end up making one query for the list and N additional queries for each item's relations.

dataloaders/index.js

import DataLoader from 'dataloader';

export const createLoaders = () => ({
  userLoader: new DataLoader(async (userIds) => {
    console.log('Batch loading users:', userIds);
    return userIds.map(id => users.find(user => user.id === id));
  }),

  userPostsLoader: new DataLoader(async (userIds) => {
    console.log('Batch loading posts for users:', userIds);
    const postsByUser = groupBy(posts, post => post.userId);
    return userIds.map(userId => postsByUser[userId] || []);
  }),

  postCommentsLoader: new DataLoader(async (postIds) => {
    console.log('Batch loading comments for posts:', postIds);
    const commentsByPost = groupBy(comments, comment => comment.postId);
    return postIds.map(postId => commentsByPost[postId] || []);
  })
});

📝 Defining the Schema

Our GraphQL schema defines the types and their relationships:

schema.js

type User {
  id: ID!
  name: String!
  email: String!
  posts: [Post!]!
}

type Post {
  id: ID!
  title: String!
  content: String!
  author: User!
  comments: [Comment!]!
}

type Comment {
  id: ID!
  text: String!
  post: Post!
  author: User!
}

🔧 Implementing Resolvers

Resolvers define how to fetch the data for each field:

resolvers.js

export const resolvers = {
  Query: {
    user: (_, { id }, { loaders }) => {
      return loaders.userLoader.load(id);
    },
    posts: (_, __, { posts }) => posts,
  },

  User: {
    posts: (parent, _, { loaders }) => {
      return loaders.userPostsLoader.load(parent.id);
    },
  },

  Post: {
    author: (parent, _, { loaders }) => {
      return loaders.userLoader.load(parent.userId);
    },
    comments: (parent, _, { loaders }) => {
      return loaders.postCommentsLoader.load(parent.id);
    },
  }
};

🌟 Example Queries

Here's how to query our API:

Query Example

query {
  user(id: "1") {
    name
    email
    posts {
      title
      content
      comments {
        text
        author {
          name
        }
      }
    }
  }
}

🎉 Benefits of This Approach

Efficient data loading with DataLoader's batching and caching
Clean separation of concerns between schema and resolvers
Type-safe API with GraphQL's type system
Flexible querying capabilities for clients

Important Note:
Remember to create new DataLoader instances for each request to prevent data leaks between different users' requests.

🔍 Performance Monitoring

You can monitor DataLoader's effectiveness by looking at the console logs. You should see multiple IDs being loaded in single batches instead of individual queries.

Pro Tip:
Use the Apollo Studio Explorer (available at http://localhost:4000) to test your queries and see the exact data being requested and returned.

2020-10-05T16:06:00.000-07:00

Spring Batch Tutorial: Processing Records with Scheduled Jobs

Learn how to implement a Spring Batch application that processes records every 5 minutes using an in-memory H2 database.

Code copied!

🎯 Project Overview

In this tutorial, we'll build a Spring Batch application that:

Uses an in-memory H2 database for storing transactions
Processes pending transactions every 5 minutes
Implements a complete batch processing pipeline
Generates sample data for testing

🛠️ Project Setup

First, let's set up our project with the required dependencies in pom.xml:

pom.xml

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-batch</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
</dependencies>

📦 Domain Model

We'll create a simple Transaction entity to represent our data:

Transaction.java

@Entity
@Data
@NoArgsConstructor
public class Transaction {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String accountId;
    private BigDecimal amount;
    private String type;
    private LocalDateTime timestamp;
    private String status;
}

⚙️ Batch Configuration

The batch job configuration is the heart of our application:

BatchConfig.java

@Configuration
@EnableBatchProcessing
@RequiredArgsConstructor
public class BatchConfig {
    @Bean
    public ItemReader reader() {
        RepositoryItemReader reader = new RepositoryItemReader<>();
        reader.setRepository(transactionRepository);
        reader.setMethodName("findByStatus");
        reader.setArguments(java.util.Arrays.asList("PENDING"));
        return reader;
    }

    @Bean
    public ItemProcessor processor() {
        return transaction -> {
            transaction.setStatus("PROCESSED");
            return transaction;
        };
    }

    @Bean
    public Job processTransactionsJob() {
        return jobBuilderFactory.get("processTransactionsJob")
                .incrementer(new RunIdIncrementer())
                .flow(processTransactionsStep())
                .end()
                .build();
    }
}

⏰ Job Scheduler

The scheduler runs our batch job every 5 minutes:

BatchJobScheduler.java

@Component
@EnableScheduling
@RequiredArgsConstructor
@Slf4j
public class BatchJobScheduler {
    @Scheduled(fixedRate = 300000) // Run every 5 minutes
    public void runBatchJob() {
        JobParameters jobParameters = new JobParametersBuilder()
                .addLong("time", System.currentTimeMillis())
                .toJobParameters();
        
        jobLauncher.run(processTransactionsJob, jobParameters);
    }
}

🔄 Sample Data Generation

We generate sample transactions on application startup:

TransactionService.java

@Service
@RequiredArgsConstructor
public class TransactionService {
    public void generateSampleTransactions() {
        for (int i = 0; i < 50; i++) {
            Transaction transaction = new Transaction();
            transaction.setAccountId(UUID.randomUUID().toString());
            transaction.setAmount(BigDecimal.valueOf(Math.random() * 1000));
            transaction.setType(Math.random() > 0.5 ? "CREDIT" : "DEBIT");
            transaction.setTimestamp(LocalDateTime.now());
            transaction.setStatus("PENDING");
            
            transactionRepository.save(transaction);
        }
    }
}

💡 Key Features

This implementation includes:

Chunk-based processing (10 items per chunk)
Automatic job parameter incrementation
Error handling and logging
Scheduled execution every 5 minutes
In-memory H2 database for easy testing

🚀 Running the Application

Clone the repository
Run mvn clean install
Start the application
Access H2 console at http://localhost:8080/h2-console
Watch the logs for batch job execution every 5 minutes

Note: The H2 database is in-memory, so data will be reset when the application restarts.

🔍 Monitoring

You can monitor the batch jobs through:

Application logs
H2 console for database inspection
Spring Batch tables (BATCH_JOB_EXECUTION, BATCH_STEP_EXECUTION)

Kafka

2018-12-31T15:48:00.000-08:00

Apache Kafka Tutorial - Java and Spark Integration

This tutorial demonstrates how to work with Apache Kafka using both Java and Apache Spark, including producer and consumer implementations with auto offset management.

Code copied!

1. Java Implementation

1.1 Kafka Producer in Java

This implementation includes idempotence and safe producer configurations for reliable message delivery.

KafkaProducerExample.java

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
import java.util.concurrent.ExecutionException;

public class KafkaProducerExample {
    private static final String TOPIC_NAME = "example-topic";
    private static final String BOOTSTRAP_SERVERS = "localhost:9092";

    public static void main(String[] args) {
        // Create producer properties
        Properties properties = new Properties();
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        
        // Create safe producer configs
        properties.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
        properties.put(ProducerConfig.ACKS_CONFIG, "all");
        properties.put(ProducerConfig.RETRIES_CONFIG, Integer.toString(Integer.MAX_VALUE));
        properties.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5");

        // Create the producer
        try (KafkaProducer producer = new KafkaProducer<>(properties)) {
            
            // Send messages with callbacks
            for (int i = 0; i < 10; i++) {
                String key = "key_" + i;
                String value = "message_" + i;
                
                ProducerRecord record = 
                    new ProducerRecord<>(TOPIC_NAME, key, value);

                producer.send(record, (metadata, exception) -> {
                    if (exception == null) {
                        System.out.printf("Message sent successfully! Topic: %s, Partition: %d, Offset: %d%n",
                            metadata.topic(), metadata.partition(), metadata.offset());
                    } else {
                        System.err.println("Error sending message: " + exception.getMessage());
                    }
                });
            }
            
            // Flush and close producer
            producer.flush();
        }
    }
}

1.2 Kafka Consumer in Java

This consumer implementation includes auto offset commit and earliest offset reset configuration.

KafkaConsumerExample.java

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class KafkaConsumerExample {
    private static final String TOPIC_NAME = "example-topic";
    private static final String BOOTSTRAP_SERVERS = "localhost:9092";
    private static final String GROUP_ID = "my-consumer-group";

    public static void main(String[] args) {
        // Create consumer properties
        Properties properties = new Properties();
        properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
        properties.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP_ID);
        properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        
        // Auto offset reset config
        properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        
        // Enable auto commit
        properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
        properties.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");

        // Create the consumer
        try (KafkaConsumer consumer = new KafkaConsumer<>(properties)) {
            // Subscribe to topic
            consumer.subscribe(Collections.singletonList(TOPIC_NAME));

            // Poll for messages
            while (true) {
                ConsumerRecords records = consumer.poll(Duration.ofMillis(100));
                
                for (ConsumerRecord record : records) {
                    System.out.printf("Received message: key = %s, value = %s, topic = %s, partition = %d, offset = %d%n",
                        record.key(), record.value(), record.topic(), record.partition(), record.offset());
                }
            }
        }
    }
}

2. Spark Implementation

2.1 Spark Kafka Producer

This Spark implementation demonstrates how to write data to Kafka using Spark's DataFrame API.

SparkKafkaProducer.scala

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object SparkKafkaProducer {
  def main(args: Array[String]): Unit = {
    // Create Spark Session
    val spark = SparkSession.builder()
      .appName("Spark Kafka Producer")
      .master("local[*]")
      .getOrCreate()

    import spark.implicits._

    // Create sample data
    val data = (1 to 100).map(i => (s"key_$i", s"message_$i"))
      .toDF("key", "value")

    // Write to Kafka
    data.selectExpr("key", "value")
      .write
      .format("kafka")
      .option("kafka.bootstrap.servers", "localhost:9092")
      .option("topic", "spark-topic")
      .save()

    spark.stop()
  }
}

2.2 Spark Kafka Consumer

This implementation uses Spark Structured Streaming to consume messages from Kafka with auto offset management.

SparkKafkaConsumer.scala

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.streaming.Trigger
import org.apache.spark.sql.functions._

object SparkKafkaConsumer {
  def main(args: Array[String]): Unit = {
    // Create Spark Session
    val spark = SparkSession.builder()
      .appName("Spark Kafka Consumer")
      .master("local[*]")
      .getOrCreate()

    import spark.implicits._

    // Read from Kafka
    val df = spark.readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", "localhost:9092")
      .option("subscribe", "spark-topic")
      .option("startingOffsets", "earliest") // Auto offset reset config
      .load()

    // Process the stream
    val query = df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
      .writeStream
      .outputMode("append")
      .format("console")
      .option("truncate", "false")
      .trigger(Trigger.ProcessingTime("1 second"))
      .start()

    query.awaitTermination()
  }
}

Dependencies

For Java Implementation:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>3.1.0</version>
</dependency>

For Spark Implementation:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
    <version>3.1.2</version>
</dependency>

2018-06-20T06:30:00.000-07:00

Building a RESTful API with Spring Boot and H2 Database

Learn how to create a production-ready REST API using Spring Boot, JPA, and H2 Database with complete CRUD operations and proper error handling.

🎯 Project Overview

We'll build a Customer Management API that includes:

CRUD operations for customer data
Input validation
Exception handling
Swagger/OpenAPI documentation
In-memory H2 database
Sample data generation

🛠️ Project Setup

First, let's set up our project with the required dependencies in pom.xml:

pom.xml

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
        <scope>runtime</scope>
    </dependency>
</dependencies>

📦 Domain Model

Create the Customer entity with validation:

Customer.java

@Entity
@Data
@NoArgsConstructor
public class Customer {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @NotBlank(message = "First name is required")
    private String firstName;

    @Email(message = "Email should be valid")
    private String email;

    @Min(value = 18, message = "Age should be at least 18")
    private Integer age;

    private Boolean active = true;
}

🔄 REST Endpoints

Method	Endpoint	Description
GET	/api/customers	Get all customers
GET	/api/customers/{id}	Get customer by ID
POST	/api/customers	Create new customer
PUT	/api/customers/{id}	Update customer
DELETE	/api/customers/{id}	Delete customer
PATCH	/api/customers/{id}/deactivate	Deactivate customer

🎯 Controller Implementation

CustomerController.java

@RestController
@RequestMapping("/api/customers")
@RequiredArgsConstructor
public class CustomerController {
    private final CustomerService customerService;

    @GetMapping
    public ResponseEntity> getAllCustomers(
            @RequestParam(required = false) Boolean activeOnly) {
        List customers = activeOnly != null && activeOnly 
            ? customerService.getActiveCustomers()
            : customerService.getAllCustomers();
        return ResponseEntity.ok(customers);
    }

    @PostMapping
    public ResponseEntity createCustomer(
            @Valid @RequestBody Customer customer) {
        return new ResponseEntity<>(
            customerService.createCustomer(customer), 
            HttpStatus.CREATED
        );
    }
}

⚡ Exception Handling

Global exception handler for consistent error responses:

GlobalExceptionHandler.java

@RestControllerAdvice
public class GlobalExceptionHandler {
    @ExceptionHandler(CustomerNotFoundException.class)
    public ResponseEntity handleCustomerNotFoundException(
            CustomerNotFoundException ex) {
        ErrorResponse error = new ErrorResponse(
            HttpStatus.NOT_FOUND.value(),
            ex.getMessage(),
            LocalDateTime.now()
        );
        return new ResponseEntity<>(error, HttpStatus.NOT_FOUND);
    }
}

📝 Sample Data

The application automatically generates sample customer data on startup:

Sample Data

{
    "firstName": "John",
    "lastName": "Doe",
    "email": "john.doe@example.com",
    "phone": "+1234567890",
    "age": 30
}

🚀 Running the Application

Clone the repository
Run mvn clean install
Start the application
Access Swagger UI at http://localhost:8080/swagger-ui.html
Access H2 Console at http://localhost:8080/h2-console

Testing the API:

You can use tools like Postman or curl to test the endpoints. For example:

curl http://localhost:8080/api/customers

Hadoop

2017-03-28T15:33:00.000-07:00

Hadoop Ecosystem Tutorial - Word Count Examples

This tutorial demonstrates how to implement word count using different components of the Hadoop ecosystem.

Code copied!

1. HDFS Example

HDFSWordCount.java

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;

public class HDFSWordCount {
    public static void main(String[] args) {
        try {
            // Initialize HDFS configuration
            Configuration conf = new Configuration();
            FileSystem fs = FileSystem.get(conf);

            // Path to input file in HDFS
            Path inputPath = new Path("/input/sample.txt");
            
            // Read file from HDFS
            FSDataInputStream in = fs.open(inputPath);
            BufferedReader reader = new BufferedReader(new InputStreamReader(in));
            
            // Word count map
            Map wordCount = new HashMap<>();
            
            String line;
            while ((line = reader.readLine()) != null) {
                String[] words = line.toLowerCase().split("\\W+");
                for (String word : words) {
                    if (word.length() > 0) {
                        wordCount.merge(word, 1, Integer::sum);
                    }
                }
            }
            
            // Write results back to HDFS
            Path outputPath = new Path("/output/hdfs_wordcount.txt");
            FSDataOutputStream out = fs.create(outputPath);
            BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(out));
            
            for (Map.Entry entry : wordCount.entrySet()) {
                writer.write(entry.getKey() + "\t" + entry.getValue() + "\n");
            }
            
            // Cleanup
            writer.close();
            reader.close();
            System.out.println("Word count completed successfully!");
            
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

2. MapReduce Example

WordCountMR.java

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
import java.util.StringTokenizer;

public class WordCountMR {
    
    public static class TokenizerMapper extends Mapper {
        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();
        
        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken().toLowerCase());
                context.write(word, one);
            }
        }
    }
    
    public static class IntSumReducer extends Reducer {
        private IntWritable result = new IntWritable();
        
        public void reduce(Text key, Iterable values, Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }
    
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "word count");
        
        job.setJarByClass(WordCountMR.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

3. PIG Example

wordcount.pig

-- Load the input file
raw_input = LOAD '/input/sample.txt' AS (line:chararray);

-- Tokenize the words
words = FOREACH raw_input GENERATE FLATTEN(TOKENIZE(LOWER(line))) AS word;

-- Group words and count occurrences
word_groups = GROUP words BY word;
word_count = FOREACH word_groups GENERATE group AS word, COUNT(words) AS count;

-- Sort results by count in descending order
ordered_word_count = ORDER word_count BY count DESC;

-- Store the results
STORE ordered_word_count INTO '/output/pig_wordcount';

4. Hive Example

wordcount.hql

-- Create a table for raw input text
CREATE TABLE IF NOT EXISTS raw_text (
    line STRING
);

-- Load data into the raw_text table
LOAD DATA INPATH '/input/sample.txt' INTO TABLE raw_text;

-- Create a table for word counts
CREATE TABLE IF NOT EXISTS word_counts (
    word STRING,
    count INT
);

-- Insert processed data into word_counts table
INSERT OVERWRITE TABLE word_counts
SELECT word, COUNT(*) as count
FROM (
    SELECT EXPLODE(SPLIT(LOWER(line), '\\W+')) as word
    FROM raw_text
    WHERE TRIM(line) != ''
) w
WHERE LENGTH(word) > 0
GROUP BY word
ORDER BY count DESC;

5. Spark Examples

Scala Implementation

WordCountSpark.scala

import org.apache.spark.sql.SparkSession

object WordCountSpark {
  def main(args: Array[String]): Unit = {
    // Create Spark session
    val spark = SparkSession.builder()
      .appName("Spark Word Count")
      .master("local[*]")  // Use local mode for testing, remove for cluster deployment
      .getOrCreate()

    // Import spark implicits for DataFrame operations
    import spark.implicits._

    try {
      // Read input file
      val inputPath = "/input/sample.txt"
      val outputPath = "/output/spark_wordcount"

      // Read text file and split into words
      val words = spark.read.textFile(inputPath)
        .flatMap(line => line.toLowerCase.split("\\W+"))
        .filter(_.nonEmpty)

      // Count words
      val wordCounts = words.groupBy("value")
        .count()
        .orderBy($"count".desc)
        .withColumnRenamed("value", "word")

      // Save results
      wordCounts.write
        .mode("overwrite")
        .format("csv")
        .option("header", "true")
        .save(outputPath)

      // Show results
      println("Word Count Results:")
      wordCounts.show()

    } finally {
      // Clean up
      spark.stop()
    }
  }
}

Python Implementation

word_count_spark.py

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode, split, lower, count, desc

def word_count_spark():
    # Create Spark session
    spark = SparkSession.builder \
        .appName("PySpark Word Count") \
        .master("local[*]") \
        .getOrCreate()

    try:
        # Input and output paths
        input_path = "/input/sample.txt"
        output_path = "/output/pyspark_wordcount"

        # Read the input file
        df = spark.read.text(input_path)

        # Process the text and count words
        word_counts = df.select(explode(split(lower(df.value), "\\W+")).alias("word")) \
            .filter("word != ''") \
            .groupBy("word") \
            .agg(count("*").alias("count")) \
            .orderBy(desc("count"))

        # Save results
        word_counts.write \
            .mode("overwrite") \
            .format("csv") \
            .option("header", "true") \
            .save(output_path)

        # Show results
        print("Word Count Results:")
        word_counts.show()

    finally:
        # Clean up
        spark.stop()

if __name__ == "__main__":
    word_count_spark()

PayPal and MVC

2011-09-14T20:25:00.000-07:00

Recently i had an opportunity to implement paypal. At first it was way too overwhelming to read about it and implement it in a day or two. However, this video link by Rob Conery cleared up lot of my confusion. I recommend seeing this video if you have any plans to implement paypal. So, PayPal SandBox is the development site where you can test your code. All the code i have implemented is either in documentation or from the video.

//Controller

public class PayPalController : Controller
{
private const string Server_URL = "https://www.sandbox.paypal.com/cgi-bin/webscr?";


public PayPalController()
{

}

public ActionResult Index()
{
MovieModel movieModel = new MovieModel();
movieModel.MovieDate = new List() {
new DateTime(2011,9,15).ToLongDateString()
};
movieModel.MovieTime = new List()
{
"12:00 PM","3:00 PM","7:00 PM","10:00 PM"
};
movieModel.NoOfTickets = new List() {1,2,3,4,5,6};
movieModel.Price = 15;
return View(movieModel);
}

public RedirectResult PostToPayPal(MovieModel movieModel)
{
string custom = GetTransactionFromDB(movieModel);
string cmd = "_xclick";
string business = "SELLEREMAIL";
string notify_url = "Localhost/PayPal/IPN";
int amount = movieModel.Price;
string item_name = "Ticket";
string redirect = string.Format("{0}&cmd={1}¬ify_url={2}&amount={3}&item_name={4}&custom={5}&business={6}",Server_URL,cmd,notify_url,amount,item_name,custom,business);
return Redirect(redirect);  
}

private string GetTransactionFromDB(MovieModel movieModel)
{
//store from model in DB and return transactionID
//store amount also,need to verify later.
return "123-234-5678";
}


public void IPN()
{
var formVals = new Dictionary();
formVals.Add("cmd", "_notify-validate");
string response = GetPayPalResponse(formVals, true);
if (response == "VERIFIED")
{
string sAmountPaid = Request["mc_gross"];
string paymentStatus = Request["payment_status"];
string customField = Request["custom"];
if (sAmountPaid == GetPriceFromDatabase(id) && paymentStatus == "Completed")
{
string buyerEmail = Request["payer_email"];
string transactionID = Request["txn_id"];//PayPal Transaction ID(Store It)
string firstName = Request["first_name"];
string lastName = Request["last_name"];
//Call to SP saying Payment Processed
}
else
{
//Call to SP Saying Payment Corrupt or Invalid
}
}
else if (response == "INVALID")
{
//Call to SP Saying Payment Failed;
}
else
{
//Send Email to Admin with response details
}
}

private string GetPriceFromDatabase(string id)
{
return "15";
}



public string GetPayPalResponse(Dictionary formVals, bool useSandbox)
{

string paypalUrl = useSandbox ? "https://www.sandbox.paypal.com/cgi-bin/webscr"
: "https://www.paypal.com/cgi-bin/webscr";


HttpWebRequest req = (HttpWebRequest)WebRequest.Create(paypalUrl);

// Set values for the request back

req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";

byte[] param = Request.BinaryRead(Request.ContentLength);
string strRequest = Encoding.ASCII.GetString(param);

StringBuilder sb = new StringBuilder();
sb.Append(strRequest);

foreach (string key in formVals.Keys)
{
sb.AppendFormat("&{0}={1}", key, formVals[key]);
}
strRequest += sb.ToString();
req.ContentLength = strRequest.Length;

string response = "";
using (StreamWriter streamOut = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII))
{

streamOut.Write(strRequest);
streamOut.Close();
using (StreamReader streamIn = new StreamReader(req.GetResponse().GetResponseStream()))
{
response = streamIn.ReadToEnd();
}
}

return response;
}


}
}

//View

<% using (Html.BeginForm("PostToPayPal", "PayPal", new { id= "Form" }))
   { %>
(Theatre Name)
Date: <%= Html.DropDownListFor(model => model.MovieDate,new SelectList(Model.MovieDate,Model.MovieDate.First())) %> <%= Html.HiddenFor(model => model.Price, new { id = "Price" })%>
<%= Html.HiddenFor(model => model.TheatreName, new { id = "TheatreName"}) %>




   Time    Tickets          Price          Buy With PayPal   
   <%= Html.DropDownListFor(model => model.MovieTime, new SelectList(Model.MovieTime, "3:00 PM"))%>    

<%= Html.DropDownListFor(model => model.NoOfTickets, new SelectList(Model.NoOfTickets, 1), new { id = "Tickets" })%>          

15$  





 
 <%} %>

Email me if you have questions. But like i said, the video has most of the explanation.

Simple Chat Prototype using ASP.Net MVC

2011-08-11T20:38:00.000-07:00

This chat application is a browser based chat.It uses ajax calls to get and send data.
On the server i have used two singletons which handle the messages and the list of online users

Controller and Models

//User Collection
namespace ChatClient.Models
{
    public sealed class UserCollection
    {

        private static UserCollection instance = null;
        private List users = new List();
        private static readonly object padlock = new object();

         UserCollection()
         {
         }

         public static UserCollection Instance
         {
                get
                {
                    lock (padlock)
                    {
                        if (instance == null)
                        {
                            instance = new UserCollection();
                        }
                        return instance;
                    }
                }
          }

         public void AddToUserCollection(User user)
         {
             users.Add(user);
         }

         public void RemoveFromUserCollection(User user)
         {
             users.Remove(user);
         }

         public bool IsUserPresent(User user)
         {
             return users.Exists(p => p.Name == user.Name);
         }
         
    }
}

public class User
    {
        public string Name { get; set; }
        public bool Online { get; set; }

    }

public class ChatMessage
    {
        public string Message { get; set; }
        public string MessageDateTime { get; set; }
        public User FromUser { get; set; }
        public User ToUser { get; set; }
    }

  public class MessageCollection
    {

         private static MessageCollection instance = null;
         private List chatMessages = new List();
        private static readonly object padlock = new object();

        MessageCollection()
         {
         }

        public static MessageCollection Instance
         {
                get
                {
                    lock (padlock)
                    {
                        if (instance == null)
                        {
                            instance = new MessageCollection();
                        }
                        return instance;
                    }
                }
          }

        public void AddToMessageCollection(ChatMessage chatMessage)
         {
             chatMessages.Add(chatMessage);
         }

        public void RemoveFromMessageCollection(User user)
         {
             chatMessages.RemoveAll(p => p.ToUser.Name == user.Name);
         }

         public List GetMessages(User user)
         {
             return chatMessages.Where(p => p.ToUser.Name == user.Name).Select(p => p).ToList();
         }

    }


//Controller
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Mvc;
using ChatClient.Models;
using System.Runtime.Remoting.Messaging;

namespace ChatClient.Controllers
{
    [HandleError]
    public class HomeController : Controller
    {
        UserCollection collection = UserCollection.Instance;
        MessageCollection messageCollection = MessageCollection.Instance;

        public ActionResult Index()
        {
            return View();
        }

        public string Register(string userName)
        {
            if (!collection.IsUserPresent(new User() { Name = userName, Online = true }))
            {
                collection.AddToUserCollection(new User()
                {
                    Name = userName,
                    Online = true,
                });
            }
            return "Online";
        }

        public JsonResult SendMessage(string message, string toUser, string fromUser)
        {
            ChatMessage chatMessage = new ChatMessage()
            {
                FromUser = new User() {
                            Name = fromUser,
                            Online = true
                },
                ToUser = new User()
                {
                    Name = toUser,
                    Online = true
                },
                Message = message,
                MessageDateTime = DateTime.Now.ToString()
            };
            messageCollection.AddToMessageCollection(chatMessage);
            return Json(chatMessage);
        }

        public JsonResult GetMessages(string userName)
        {
            if (messageCollection.GetMessages(new User() { Name = userName, Online = true }).Count > 0)
            {
                List messages = messageCollection.GetMessages(new User() { Name = userName, Online = true }).Select(p => p).ToList();
                Action remove = delegate()
                {
                    RemoveMessages(userName);
                };
                IAsyncResult result = remove.BeginInvoke(RemoveMessagesEnd, null);
                return Json(messages.ToArray(),JsonRequestBehavior.AllowGet);
            }
            return Json(string.Empty, JsonRequestBehavior.AllowGet);
        }
        public void RemoveMessagesEnd(IAsyncResult result)
        {
            object state = (AsyncResult)result.AsyncState;
        }

        public void RemoveMessages(string userName)
        {
            messageCollection.RemoveFromMessageCollection(new User() { Name = userName, Online = true });
        }
        public ActionResult About()
        {
            return View();
        }
    }
}

View


  
  <%= Html.Label("UserName") %>
 <%= Html.TextBox("userName")%>
 <%= Html.Label("ToUser") %>
 <%=Html.TextBox("toUser") %>
 
 <%= Html.Label("ChatHistory") %>
 <%= Html.TextArea("ChatHistory")%>
 <%= Html.Label("ChatBox") %>
 <%= Html.TextBox("ChatBox") %>

Jqgrid for MVC 2

2011-05-25T08:43:00.000-07:00

Here i will put in code to demonstrate Jqgrid.This is one of the coolest grids available.There are many ways to implement this grid.There is a lot of documentation available for Jqgrid on its wiki,so i am not going to explain the stuff.The way i have implemented the grid is to have a search form and have ajax call made to the search controller and then let it return us a array data and that we feed directly to the grid.This grid will also have a double click event hooked up for all the rows returned.I have also used Block UI to block the page when we construct a dialog with all the information about the record.One problem i faced with Json is,the default max for JSon Serialization is very small,so i found a class LargeJSon on google which basically increases the default limit to

<jsonSerialization maxJsonLength="2147483644""></jsonSerialization>.

Here is the Main code

<script type="text/javascript" language="javascript">
$(document).ready(function() {


function Search() {
$(loading).show();
$.ajax({
type: "POST",
url: '<%=Url.Content("~/Search/Search")%>',
data: $("#SearchForm").serialize(),
datatype: 'json',
success: function(msg) {
if (msg != null) {
$("#Validation").hide();
var gridData = msg;
if (!loaded) {
loaded = LoadGridWithData(gridData);
}
else {
$("#list").GridUnload();
loaded = LoadGridWithData(gridData);
}
$(loading).hide();
}
},
error: function(xhr, status, error) {
var object = new Array();
object = xhr.responseText.split(';');
var response = '';
for (var i = 0; i < object.length; i++) {
var p = "<div>" + object[i] + "</div>";
response += p;
}
$("#Validation").html(response);
$("#Validation").show();
$(loading).hide();
}
});

}

var dialogOpen = false;

function LoadGridWithData(gridData) {
jQuery("#list").jqGrid({
sortable: true,
data: gridData,
datatype: 'local',
mtype: 'GET',
colNames: ['First Name', 'Middle Name', 'Last Name', 'Agent', 'State', 'Zip', 'Number', 'Type', 'Status', 'Date'],
colModel: [
{ name: 'FirstName', index: 'FirstName', width: 30, sorttype: 'text' },
{ name: 'MiddleName', index: 'MiddleName', width: 15, sorttype: 'text', align: 'left' },
{ name: 'LastName', index: 'LastName', width: 40, sorttype: 'text', align: 'left' },
{ name: 'Agent', index: 'Agent', width: 30, sorttype: 'int', align: 'left' },
{ name: 'State', index: 'State', width: 15, sorttype: 'text' },
{ name: 'Zip', index: 'Zip', width: 20, sorttype: 'int', align: 'left' },
{ name: 'Number', index: 'Number', width: 50, sorttype: 'int', align: 'left' },
{ name: 'Type', index: 'Type', width: 50, sorttype: 'text', align: 'left' },
{ name: 'Status', index: 'Status', width: 50, sorttype: 'text', align: 'left' },
{ name: 'Date', index: 'Date', width: 50, sorttype: 'date', resizable: false }
],
pager: '#pager',
rowNum: 20,
rowList: [5, 10, 20, 50],
gridview: true,
sortname: 'Number',
sortorder: "desc",
height: 400,
loadui: "enable",
loadtext: "Loading...",
autowidth: true,
rownumbers: true,
viewrecords: true,
caption: 'Quote Header Data',
ondblClickRow: function(id) {
$.blockUI({ message: "<h1>Retrieving Information. Please Wait...</h1>", timeout: 10000 });
var url = '<%= Url.Content("~/") %>' + "Search/Modal";
$.post(url, { Id: id }, function(response, status, xhr) {
if (status != "error") {
$.unblockUI();
$('#verificationedit-form').html(response).dialog("open");
}
else {
$.blockUI({ message: "<h1>Retrieving Failed.Please Try Again...</h1>" });
$('.blockOverlay').attr('title', 'Click to unblock').click($.unblockUI);
}
});
}
});
jQuery("#list").jqGrid('navGrid', '#pager', { del: false, add: false, edit: false, search: false });
return true;
}
}
</script>



<table id="list" cellpadding="0" cellspacing="0">
</table>
<div id="pager" style="text-align: center;">
</div>
<div id="verificationedit-form" title="Edit Verification Information">
</div>

//Controller

[HttpPost]
[EnableCompression]
public JsonResult Search(SearchModel searchModel)
{
LargeJsonResult result = new LargeJsonResult();

if (ModelState.IsValid)
{
List<GridModel> model = repository.Search(searchModel);
var jsonData =
(from pp in model
select new
{
id = pp.Id,
pp.FirstName,
pp.MiddleName,
pp.LastName,
pp.Agent,
pp.State,
pp.Zip,
pp.Number,
pp.Type,
pp.Status,
pp.Date
}).ToArray();

result.Data = jsonData;
result.JsonRequestBehavior = JsonRequestBehavior.AllowGet;
return result;
}
else
{
StringBuilder errorString = Utilities.BuildErrorMessageFromModelStateWithDelimiter(ModelState, ";");
Response.StatusCode = 500;
Response.Write(errorString.ToString());
return result;
}
}

Next time i will post the Jggrid code with inline editing.Happy Coding!!

autocomplete

2011-01-26T15:26:00.000-08:00

autocomplete is one the coolest features for a UI to have.So i happened to get a chance to make it.I had some struggle initially as i am new to jquery but finally got it and it works like a charm.

To do this you need to download Jquery autocomplete bundle.So,all we need to do is when the document loads we need to do a ajax call to your action to get your results(which is coming from your database or file,etc).For me it is the database,so you want to play around with it to find a sweet spot where your UI is not getting a performance hit because your query is taking too long.

jQuery(document).ready(function() {
$.ajax({
type: "GET",
url: "controller/AutoComplete",
data: '{}',
success: function(msg) {
// Do something interesting here.
if (msg != null) {
data = msg.toString().split(",");
$("#Name").autocomplete(data);
}
},
error: function(xhr, status, error) {
// Boil the ASP.NET AJAX error down to JSON.
var err = eval("(" + xhr.responseText + ")");
// Display the specific error raised by the server
alert(err.Message);
}
});
});

Now in your action,you need to call your repository to return back results,here i am returning a comma seperated array which i spit it and feed to the the autocomplete script.

[HttpGet]
public ActionResult AutoComplete()
{
string model = repository.GetDistinctNames();
return Content(model);
}

That's it you see autocomplete feature in your UI and your clients will be happy.

FileSystemWatcher Using Rx

2010-10-14T13:23:00.000-07:00

This is a simple program which checks and reports for error files produced by ETL jobs.

namespace ErrorFileWatcher
{
public class ErrFileWatcher
{
private readonly FileSystemWatcher _fileSystemWatcher;
public IObservable ChangedFiles { get; set; }
public IObservable CreatedFiles { get; set; }
public IObservable DeletedFiles { get; set; }
public IObservable RenamedFiles { get; set; }
public IObservable Errors { get; set; }


public ErrFileWatcher(string directory, string filter, bool includeSubdirectories)
{
_fileSystemWatcher = new FileSystemWatcher(directory, filter)
{
EnableRaisingEvents = true,
IncludeSubdirectories = includeSubdirectories
};

ChangedFiles = Observable.FromEvent(
(EventHandler e) => new FileSystemEventHandler(e),
e => _fileSystemWatcher.Changed += e,
e => _fileSystemWatcher.Changed -= e).Select(x => x.EventArgs);

CreatedFiles = Observable.FromEvent(
(EventHandler e) => new FileSystemEventHandler(e),
e => _fileSystemWatcher.Created += e,
e => _fileSystemWatcher.Created -= e).Select(x => x.EventArgs);


DeletedFiles = Observable.FromEvent(
(EventHandler e) => new FileSystemEventHandler(e),
e => _fileSystemWatcher.Deleted += e,
e => _fileSystemWatcher.Deleted -= e).Select(x => x.EventArgs);


RenamedFiles = Observable.FromEvent(
(EventHandler e) => new RenamedEventHandler(e),
e => _fileSystemWatcher.Renamed += e,
e => _fileSystemWatcher.Renamed -= e).Select(x => x.EventArgs);



Errors = Observable.FromEvent(
(EventHandler e) => new ErrorEventHandler(e),
e => _fileSystemWatcher.Error += e,
e => _fileSystemWatcher.Error -= e).Select(x => x.EventArgs);


}

}
}

Windows Service code which calls the above class and also calls the sendemail method

try
{

while (true)
{
if (!RunToday)
{
if (IsItTimeToRun())
{
EventLog.WriteEntry("ErrorFileWactcher", "Watcher Started For " + DateTime.Now.Date, EventLogEntryType.Information);
SendEmails emails = new SendEmails();
Stopwatch sW = new Stopwatch();
sW.Start();
TextWriter writer = new StreamWriter(ConfigurationManager.AppSettings["FilePath"].ToString());
ErrFileWatcher fileWatch = new ErrFileWatcher(ConfigurationManager.AppSettings["FILEPATHTOBEMONITORED"], "*.txt", false);
fileWatch.CreatedFiles.Where(p => p.FullPath.Length > 0).Select(p => p.Name).Subscribe(p => writer.WriteLine(p.ToString() + " Created"));
fileWatch.ChangedFiles.Where(p => p.FullPath.Length > 0).Select(p => p.Name).Subscribe(p => writer.WriteLine(p.ToString() + " Changed"));
fileWatch.DeletedFiles.Where(p => p.FullPath.Length > 0).Select(p => p.Name).Subscribe(p => writer.WriteLine(p.ToString() + " Deleted"));
fileWatch.RenamedFiles.Where(p => p.FullPath.Length > 0).Select(p => p.Name).Subscribe(p => writer.WriteLine(p.ToString() + " Renamed"));
while (true)
{
if (sW.Elapsed.Minutes > 10)
{
sW.Stop();
writer.Flush();
writer.Dispose();
emails.SendEmail(true, ConfigurationManager.AppSettings["FilePath"].ToString(), "ERRORREPORT");
break;
}
}
}
else
{
Thread.Sleep(int.Parse(ConfigurationManager.AppSettings["SleepTime"]));
}
}
else
{
Thread.Sleep(int.Parse(ConfigurationManager.AppSettings["SleepTime"]));
}
}
}
catch (Exception ex)
{
EventLog.WriteEntry("ErrorFileWactcher", ex.Message, EventLogEntryType.Error);
}

Rx sample

2010-08-25T08:55:00.000-07:00

In this post i have implemented a simple program which writes all the information of a customer class to a file.I took this example as it can be functional in a very short period of time.The code in this blog is inspired from Rx wiki.

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Linq;
using System.Text;
using System.Collections.Specialized;
using System.Configuration;
using System.IO;
using System.Threading;
using System.Concurrency;
using System.Diagnostics;

namespace ObservablePrototype
{
public class Prototype
{
string errorfile = ConfigurationManager.AppSettings["CustomerFile"];

public void RunProto()
{
Stopwatch sw = new Stopwatch();
sw.Start();

//Making Customer as observableCollection 
var customers = new ObservableCollection();

// adding event handler 
var customerChanges = Observable.FromEvent(
(EventHandler ev)
=> new NotifyCollectionChangedEventHandler(ev),
ev => customers.CollectionChanged += ev,
ev => customers.CollectionChanged -= ev);

//ling query for notify collection changed

var line = (from c in customerChanges
where c.EventArgs.Action == NotifyCollectionChangedAction.Add
from cu in c.EventArgs.NewItems.Cast().ToObservable()
select cu).BufferWithCount(10);

//using scheduler or u can just subscribe directly
Scheduler.ThreadPool.Schedule(() =>
{
line.Subscribe(cus =>
{
WriteCustomers(cus.ToList());
}
);
});

//start adding customers  
AddCustomers(customers);
WriteTimings(sw.Elapsed.Seconds + "Observable");
}

private void WriteTimings(string p)
{
TextWriter writer = new StreamWriter(errorfile, true);
writer.WriteLine(p);
writer.Close();
}



private void AddCustomers(ObservableCollection customers)
{
for (int i = 0; i < 50; i++)
            {
                Thread.Sleep(500);
                customers.Add(new Customer() { firstName = "K", lastName = "P" + i });
            }
            
        }



        public void StraightWrite()
        {
            Stopwatch sw = new Stopwatch();
            sw.Start();
            List customers = new List();
for (int i = 0; i < 50; i++)
            {
                Thread.Sleep(500);
                customers.Add(new Customer() { firstName = "K", lastName = "P" + i });
            }
            WriteCustomers(customers);
            WriteTimings(sw.Elapsed.Seconds + "Normal");
        }



        private void WriteCustomers(List customers)
{
foreach (var c in customers)
{
TextWriter writer = new StreamWriter(errorfile, true);
writer.WriteLine("FirstName : " + c.firstName + "  LastName : " + c.lastName);
writer.Close();
}
}

}
}

OTAClient + QC reporting with C#

2010-07-14T21:58:00.002-07:00

When writing a Data Comparison tool, one of the requirements was to log the errors into Quality Center. Thats where OTAClient.dll comes into picture.
Open test Architecture is provided to access QC's database without using its GUI.
The code block below populates the bug object,posts it in QC's DB and also sends email to the concerned person.

public void GetDistinctRowsForQC()
{
DataContext dataContext = new DataContext();

var qc = (from details in dataContext.TableName
where details.RunId == _testRunID
select details);

QCReporting.LogErrorInQualityCenter(qc.Take(2));
}


internal static void LogErrorInQualityCenter(IQueryable qc)
{
TDConnectionClass QCConnection = new TDConnectionClass();
QCConnection.InitConnectionEx("http://qualitycenter/qcbin/");
QCConnection.Login("UserID", string.Empty);
QCConnection.Connect("C", "C");

BugFactory BFactory = (BugFactory)QCConnection.BugFactory;
Customization field = (Customization)QCConnection.Customization;
CustomizationFields f = (CustomizationFields)field.Fields;

List list = f.get_Fields("Bug");
Dictionary columnnames = new Dictionary();
foreach (CustomizationField fieldd in list)
{
if(!columnnames.ContainsKey(fieldd.UserLabel))
columnnames.Add(fieldd.UserLabel,fieldd.ColumnName);
}
int i = 0;
foreach(var q in qc)
{
IBug bug = (IBug)BFactory.AddItem(i);
bug.AutoPost = false;
bug.AssignedTo = "UserID";
bug.DetectedBy = Environment.UserName;
bug.Priority = "1-Low";
bug.Status = "New";
bug.Summary = q.ErrorMessage;
bug.Project = "CCC";
bug["BG_Reproducible"] = "Y";
bug["BG_Severity"] = "2-Medium";
bug[columnnames.Where(P => P.Key == "Issue Type").Select(p => p.Value).FirstOrDefault()] = "Defect";
bug[columnnames.Where(P => P.Key == "SDLC Phase").Select(p => p.Value).FirstOrDefault()] = "2-Project Execution";
bug[columnnames.Where(P => P.Key == "Detected on Date").Select(p => p.Value).FirstOrDefault()] = DateTime.Now;
bug[columnnames.Where(P => P.Key == "Application Group").Select(p => p.Value).FirstOrDefault()] = "Cu";
bug[columnnames.Where(P => P.Key == "Application Sub Group").Select(p => p.Value).FirstOrDefault()] = "CA";
bug[columnnames.Where(P => P.Key == "Project Number").Select(p => p.Value).FirstOrDefault()] = "C";
bug[columnnames.Where(P => P.Key == "Environment").Select(p => p.Value).FirstOrDefault()] = "QA";
bug[columnnames.Where(P => P.Key == "Detector Name").Select(p => p.Value).FirstOrDefault()] = new DirectoryEntry("WinNT://p/" + Environment.UserName + "").Name;
bug[columnnames.Where(P => P.Key == "Assigned Name").Select(p => p.Value).FirstOrDefault()] = new DirectoryEntry("WinNT://p/" + Environment.UserName + "").Name;
bug[columnnames.Where(P => P.Key == "Verified By").Select(p => p.Value).FirstOrDefault()] = "UserID";
bug[columnnames.Where(P => P.Key == "Verifier Name").Select(p => p.Value).FirstOrDefault()] = new DirectoryEntry("WinNT://p/" + Environment.UserName + "").Name;
bug[columnnames.Where(P => P.Key == "ReopenCountField").Select(p => p.Value).FirstOrDefault()] = "0";
bug[columnnames.Where(P => P.Key == "Source").Select(p => p.Value).FirstOrDefault()] = "Test";
bug[columnnames.Where(P => P.Key == "Reopen Count").Select(p => p.Value).FirstOrDefault()] = "0";
bug.Post();

bug.Mail("UserID", "UserID", 4, "New Defect Notification", q.ErrorMessage);
i++;
}
QCConnection.DisconnectProject();
QCConnection.Logout();
}

Time	Tickets	Price	Buy With PayPal
<%= Html.DropDownListFor(model => model.MovieTime, new SelectList(Model.MovieTime, "3:00 PM"))%>	<%= Html.DropDownListFor(model => model.NoOfTickets, new SelectList(Model.NoOfTickets, 1), new { id = "Tickets" })%>	15$