搜斧

我收藏的链接（40）

Sun, 05 Jul 2009 00:12:00 GMT

InfoQ: 如何在团队中引入和评估代码质量: * 要写好的代码需要学习什么？ * 我如何评估眼前的代码质量是好还是坏？ * 代码好坏的因素或者标准是什么？ * 如何让代码解决质量问题？ * 如何对糟糕的代码进行优化？
Apache2.2和SVN1.4.4搭建svn版本控制平台（windows环境） - 志权黄的日志 - 网易博客: Apache2.2和SVN1.4.4搭建svn版本控制平台（windows环境）
5. GWC - GeoWebCache - GeoServer: GeoWebCache is a WMS tiling client. It runs as a proxy server between a map client and map server, caching tiles as they are requested, thus saving vast amounts of processing time. Written in Java, GeoWebCache is available as a standalone product which can be integrated with many client/server combinations. However, GeoWebCache is also an extension that has been shipped with GeoServer since version 1.7.0.
TileCache Tutorial - GeoServer: In the GeoServer community we've been really enjoying MetaCarta's TileCache as a caching layer in front of the WMS. You can see it in action at http://sigma.openplans.org. If you have non-dynamic mapping data in GeoServer using it will help give your users an experience that comes close to matching Google Maps in terms of responsiveness and usability. This tutorial will take you through setting it up (mostly pointing at their docs) as well as some hints and tricks that we've come upon in working with it. If you have information we've missed her don't hesitate to include it, this is a wiki, just hit 'page operations -> edit page' on the left. We're currently recommending this solution over squid or oscache, as TileCache was specially designed for WMS.
BIRT Example Reports: This section of Examples focuses on the feature set within BIRT. Simple reports are used to illustrate a function within BIRT. Most examples will have a before and after report that can be downloaded. In addition, the example is illustrated in a Flash presentation. The examples use popups to explain the process and provide no sound.
Eclipse BIRT报表制作演示视频: Eclipse BIRT报表制作演示视频，含图表
安装ImageMagick和JMagick的注意事项 | 姓张的小子: Linux安装ImageMagick和JMagick的注意事项
ImageMagick, JMagick安装、配置及使用 - 魏杰的技术专栏 - 51CTO技术博客: Pure java的图片操作，不依赖操作系统的功能，图片运算量比较大，如果批处理图片或图片size较大的话（如1M左右的图片）的话，性能肯定不能满足；用JNI(java本地调用)，依赖操作系统提供的功能，操作批量、大图片，就成为首选方案。
使用 Drools 规则引擎实现业务逻辑: 使用规则引擎可以通过降低实现复杂业务逻辑的组件的复杂性，降低应用程序的维护和可扩展性成本。这篇更新的文章展示如何使用开源的 Drools 规则引擎让 Java™ 应用程序更适应变化。Drools 项目引入了一个新的本地规则表达式语言和一个 Eclipse 插件，使 Drools 比以前更容易使用
Geomajas | Edit and integrate GIS data from within your browser: Geomajas is an Open source Web mapping software framework enabling 'real integrated GIS via the browser'. Geomajas includes: * Geometry and attribute editing with snapping support * Complex attribute and relation mapping (one-to-many, many-to-one), helping you implement business logic * Advanced querying capabilities (searching, filters, styles, ...)
BIRT：基于 Eclipse 的报表: BIRT：基于 Eclipse 的报表，教程，入门，快速上手
Project Voldemort Blog : News about the project: Building Voldemort read-only stores with Hadoop
Pentaho Commercial Open Source Business Intelligence: Mondrian is an OLAP server written in Java. It enables you to interactively analyze very large datasets stored in SQL databases without writing SQL.
Using Location-Enabled MIDlets for Mobile Navigation: Location information has become fairly easy to access nowadays, even on mobile devices. The widespread availability of GPS chipsets like the SiRF chipsets, which are accurate to within a few meters, as well as the significantly lower costs of these products, have contributed to the accessibility. The Location API for J2ME (JSR 179) even makes it very easy for developers to integrate location information into mobile applications.
XTAPI JTAPI: The JTAPI specification has been out for several years with no free working implementation. The wait is over. XTAPI provides a working implementation of the Core and Media packages for JTAPI 1.2.
Java开源网络服务器端组件分类列表: http，socket，ftp，代理服务器开源项目
iReport图表(Chart)教程: iReport很多的图表(Chart)类型可提供使用。iReport现在支持Pie,Pie 3D,Bar,Bar 3D,XY Bar,Stacked Bar,Stacked Bar 3D,Line,XY Line,Area,XY Area,Scatter Plot,Bubble,Time Series,High Low Open Close和Candlestick 多种类型图表。 iReport使用jfreechart作为其图表引擎，jfreechart是一个强大的chart 生成工具。
利用iReport进行图表开发-苹果爸爸-搜狐博客: 以柱状图（bar3DChart）为例，详细说明图表的开发过程。
用 BIRT 实现高级图表: BIRT 提供了一个图表引擎，使我们能够把数据集显示为图表。还可以进一步配置图表来支持交互操作，这样的话，当用户在图表元素（比如图表标题、图例、轴、值系列等）上执行操作时，图表就会做出各种响应，比如显示工具提示、弹出消息或跳到另一个报告。BIRT 可以显示以下格式的图表：PNG、JPG、BMP 或 SVG。SVG 是一种描述 2-D 矢量图形和脚本编程的 XML 规范，因此可以实现各种交互式特性：钻通允许用户通过单击行、节点等图表元素从一个报告转到另一个报告。这样就可以通过一个选择的元素把两个报告链接起来
InfoQ: Web自动化测试工具Selenium 1.0正式发布:Chrome支持+用户指南: Selenium是一款开源Web自动化测试工具，最近发布了1.0版，标志着Web自动化测试领域正式加入了一名新成员。在其新版本中，除了修正了若干 Bug，最引人瞩目的就是Selenium RC增加了对Google Chrome浏览器的支持，同时Selenium官方网站上提供了完整的用户指南
http://dev.openstreetmap.org/~edgemaster/OpenLayersExamples/POI/: OpenLayers Examples 显示点的标注
- Sky.Jian 朝阳的天空: 简朝阳 (Sky Jian)博客 MySQL & Oracle DBA 目前就职于阿里巴巴
如何用javascript在页面上注册activex事件的处理函数 - 天涯问答: 如何用javascript在页面上注册activex事件的处理函数
vb开发的activex 与javascript 交互 - 王朝网络 - wangchao.org: vb开发的activex 与javascript 交互
在用c#开发的ActiveX中调用JavaScript方法 - C# - 拼吾爱程序人生最新编程技术交流社区: 如何在ActiveX中调用页面上的JavaScript方法。
Recent News Open Source Flash: OSFlash is a watering hole for open source Flash developers and users to meet, talk about and contribute to new and existing open source projects for the Flash Platform.
InfoQ: 热衷敏捷测试的十大理由: # 不再需要手工测试脚本！－相反，自动运行的脚本让测试人员有更多的时间来挖掘测试。 # 开发人员喜欢我了！－迭代结束之前发现问题，而且因为开发人员对代码还有一个比较清晰的印象，所以比较易于找到问题。 # 现在我可以在撰写特性之前就分解特性！（Kay 与 Philip）－在撰写特性之前开始测试，测试人员可以预防问题。 # 自动化测试在一天之内运行很多次－任何修改都能得到快速反馈。 # 营造团队导向的氛围－（John Overbaugh）－每位团队成员不仅关心编码，也会关心测试是否完成（Lisa Crispin） # 测试人员可以解决偶发性bug（Lisa Crispin）－自动化的测试让每个人都舒服。 # 经常复审测试实践的机会（Adam Knight）－不再是对过去行为的简单重复，实践经常会被复审。在 Adam 的例子里面，过去要5天完成的手工测试减少到只需要30分钟。 # 我只花很少、很少的时间来调试（Adrian Howard）－当我犯了错，我能很快得到反馈－所以轻而易举就找到问题，然后解决。 # 真正改进质量，而不是仅仅记录在文档上（John Overbaugh）－ bug很快就被解决，而不是只放在bug表里面。 # 因为测试先行，测试的时间总是有的－ Josue Barbosa dos Santos 讲述了在巴西的一个政府办公室工作的故事，那里测试被安排在项目的最后阶段。开发工作总是落后于项目时间表，面临截止期限的项目不测试就发布给用户。引入 TDD和ATDD之后，最少有一部分测试会随着软件开发同步进行。
F5 BIG-IP负载均衡器配置实例与Web管理界面体验[原创] - 回忆未来[张宴] - 服务器系统架构与底层研发: F5 BIG-IP负载均衡器配置实例与Web管理界面体验
Spring Roo provides interactive, lightweight, user customizable tooling that enables rapid delivery of high performance enterprise Java applications: Spring Roo provides interactive, lightweight, user customizable tooling that enables rapid delivery of high performance enterprise Java applications. Roo is currently in early development, with http://www.springsource.org/roo representing the project's temporary project home page. We'll add more content very soon!
SQL经典面试题及答案 - - JavaEye技术网站: SQL经典面试题及答案
数据库面试题集合 - - JavaEye技术网站: 数据库面试题集合
java面试题集 - - JavaEye技术网站: java面试题集
Home - HermesJMS - Confluence: HermesJMS is an extensible console that helps you interact with JMS providers making it easy to browse or seach queues and topics, copy messages around and delete them. It fully integrates with JNDI letting you discover administered objects stored, create JMS sessions from the connection factories and use any destinations found. Many providers include a plugin that uses the native API to do non-JMS things like getting queue depths (and other statistics) or finding queue and topic names.
解决：服务器无法加载应用程序 '/LM/W3SVC'。: 浏览ASP页面提示 500内部错误
Monit: Monit is a free open source utility for managing and monitoring, processes, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.
InfoQ: Javier Paniza: Java开发效率也可以比RoR高: 为了证明他的观点，他开始用带JPA注解的POJO，以及模型驱动框架OpenXava，实现了Ruby文章中的一个应用，结果是用了更少的代码，更少的时间却获得一个更加强大的应用。在用户界面、负责CRUD的controller、实体关系维护等方面，Java方案均表现突出。
lucene HitCollector 的作用 - 信息检索与算法研究 - JavaEye技术网站: lucene HitCollector 的作用
The Software Cookbook - Improving performances of a Lucene Search: Improving performances of a Lucene Search
SourceForge.net: Fast Image Resizer for Java - Project Web Hosting - Open Source Software: Fast Image Resizer for Java Tool是一个能够批量调整同一个文件中JPEG/JPG图片大小的Java Tool。支持将调整后的图片存成ZIP文件。为图片添加文本或图片水印。自动生成Web相册。可以设置输出图片的质量，重命名创建的文件（比如全部改成小写，添加前缀，或按某一规则顺序命名）。
InfoQ: 提升可伸缩性的八项最佳实践: 提升可伸缩性的八项最佳实践
逆水行舟 » Blog Archive » 对apache timeout参数的思考: apache timeout参数解释

阅读全文...

用jdbc代替hibernate处理大批量的关联数据更新

Sun, 28 Jun 2009 08:50:00 GMT

在Spring+hibernate大批量数据处理描述了可以用hibernate快速的处理大批量数据更新：

14.2. 批量更新（Batch updates）

此方法同样适用于检索和更新数据。此外，在进行会返回很多行数据的查询时，你需要使用 scroll() 方法以便充分利用服务器端游标所带来的好处。

Session session = sessionFactory.openSession();
Transaction tx = session.beginTransaction();
   
ScrollableResults customers = session.getNamedQuery("GetCustomers")
    .setCacheMode(CacheMode.IGNORE)
    .scroll(ScrollMode.FORWARD_ONLY);
int count=0;
while ( customers.next() ) {
    Customer customer = (Customer) customers.get(0);
    customer.updateStuff(...);
    if ( ++count % 20 == 0 ) {
        //flush a batch of updates and release memory:
        session.flush();
        session.clear();
    }
}
   
tx.commit();
session.close();

条件是：
1，将JDBC的批量抓取数量（batch size）参数设置到一个合适值 （其实是500——1000比较合适）： 
hibernate.jdbc.batch_size 20

2，在执行批量处理时关闭二级缓存：

hibernate.cache.use_second_level_cache false

但是要是在做了上述操作后，还需要对这些记录关联的另外一个表的记录做一个更新，那用hibernate是有问题的。
因为Hibernate更新一条记录的时候必须首先load查出这条记录，那在批量更新的时候不是很耗费性能？确实是这样的。
所以，在大批量处理更新一个大型数据集合，并且需要修改这个集合关联的数据库表对象时，还是需要JDBC来解决问题。如：

        Connection conn1 = this.getConnection();
        PreparedStatement pstmt1 = null;
        ResultSet rs1 = null;
        String strsql1 = "select workitem_id,work_sheet_no,process_code,process_name,process_ins_id,"
            +"activity_code,activity_name,zone_code,activity_role,run_mode,pre_activity_code,"
            +"pre_activity_name,sendor,sendor_name,participant,participant_name,approve_note,"
            +"start_time,end_time,time_limit,time_limit_unit,over_time,reassign_participant_id,"
            +"form_type,form_id,app_name,urge_date,urger,last_date,last_count,untread,state,remark,"
            +"activity_role_id from wf_workitem r where r.process_code='ec_reprints_invoice_process'";

        Connection conn2 = this.getConnection2();
        PreparedStatement pstmt2 = null;
        String strsql2 = "insert into os_currentstep(id,entry_id,step_id,action_id,owner,start_date,"
            +"finish_date,due_date,status,caller) values(seq_os_currentsteps.nextval,?,?,?,?,?,?,?,?,?)";
        int updateNum=0;
        try {
            pstmt1 = conn1.prepareStatement(strsql1);
            pstmt1.setFetchSize(1000);

            pstmt2 = conn2.prepareStatement(strsql2);

            rs1 = pstmt1.executeQuery();
            int i=0;
            while (rs1.next()) {
                i++;
                String process_ins_id = rs1.getString("process_ins_id");
                String idsub=process_ins_id.substring(process_ins_id.indexOf("/", 1));

                pstmt2.setObject(1, idsub.substring(1,idsub.indexOf("/", 1)));
                pstmt2.setObject(2, getStepIdByStepName(rs1.getString("activity_name"),rs1.getString("process_code")));
                pstmt2.setNull(3,Types.INTEGER);
                pstmt2.setObject(4, rs1.getString("activity_role"));
                Date d=convertStringToDate("yyyy-MM-dd HH:mm:ss", rs1.getString("start_time"));
                if(d==null){
                    pstmt2.setNull(5,Types.DATE);
                }else{
                    pstmt2.setObject(5, new java.sql.Timestamp(d.getTime()));
                }
                pstmt2.setNull(6,Types.DATE);
                pstmt2.setNull(7,Types.DATE);
                if(rs1.getString("participant")==null){
                    pstmt2.setObject(8,"Queued");
                }else{
                    pstmt2.setObject(8,"Underway");
                }
                pstmt2.setObject(9, rs1.getString("participant"));

                pstmt2.addBatch();
                if(i%500==0){
                    updateNum=updateNum+pstmt2.executeBatch().length;
                }
            }
            updateNum=updateNum+pstmt2.executeBatch().length;
            System.out.println("更新了当前待办记录条数："+updateNum);
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            cleanConn(conn1,pstmt1,rs1);
            cleanConn(conn2,pstmt2,null);
        }

为了健康，将你的身体从酸性变为碱性

Sun, 07 Jun 2009 05:52:33 GMT

为了健康，将你的身体从酸性变为碱性

与碱性体质者相比，酸性体质的人常会感到身体疲乏、记忆力减退、注意力不集中、腰酸腿痛、腹泻、便秘等，到医院也检查不出什么毛病。如果长期处在酸性体质不加以改善，女性的皮肤会过早地黯淡和衰老；儿童会造成发育不良、食欲不振、注意力难以集中等症状；中老年人则会因此引发糖尿病、神经系统疾病和心脑血管疾病。85％的痛风、高血压、癌症、高脂血症患者，都是酸性体质。

常见食物的酸碱性

强酸性蛋黄、乳酪、白糖、西点、柿子、乌鱼子、柴鱼等。

中酸性火腿、鸡肉、鲔鱼、猪肉、鳗鱼、牛肉、面包、小麦、奶油、马肉等。

弱酸性白米、花生、啤酒、油炸豆腐、海苔、文蛤、章鱼、泥鳅等。

弱碱性红豆、萝卜、苹果、甘蓝菜、洋葱、豆腐等。

中碱性萝卜干、大豆、胡萝卜、番茄、香蕉、橘子、番瓜、草莓、蛋白、梅干、柠檬、菠菜等。

强碱性葡萄、茶叶、葡萄酒、海带、天然绿藻类

开源的Java ETL (Extraction, Transform, Load) 工具

Sat, 30 May 2009 03:27:28 GMT

ETL，是英文 Extract-Transform-Load 的缩写，用来描述将数据从来源端经过萃取（extract）、转置（transform）、加载（load）至目的端的过程。ETL一词较常用在数据仓库，但其对象并不限于数据仓库。
ETL所描述的过程，一般常见的作法包含ETL或是ELT(Extract-Load-Transform)，并且混合使用。通常愈大量的数据、复杂的转换逻辑、目的端为较强运算能力的数据库，愈偏向使用ELT，以便运用目的端数据库的平行处理能力。

ETL(or ELT)的流程可以用任何的编程语言去开发完成，由于ETL是极为复杂的过程，而手写程序不易管理，有愈来愈多的企业采用工具协助ETL的开发，并运用其内置的metadata功能来储存来源与目的的对应（mapping）以及转换规则。

工具并可以提供较强大的连接功能（connectivity）来连接来源及目的端，开发人员不用去熟悉各种相异的平台及数据的结构，亦能进行开发。

下面是一些开源的Java ETL工具

Octopus - Octopus is a simple Java-based Extraction, Transform, and Loading (ETL) tool. It may connect to any JDBC data sources and perform transformations defined in an XML file. A loadjob-generator is provided to generate Octopus loadjob skeletons from an existing database. Many different types of databases can be mixed (MSSQL, Oracle, DB2, QED, JDBC-ODBC with Excel and Access, MySQL, CSV-files, XML-files,...) Three special JDBC drivers come with Octopus to support JDBC access to CSV-files (CSV-JDBC), MS-SQL (FreeTDS) and XML. Octopus supports Ant and JUnit to create a database / tables and extract/load data during a build or test process.

Xineo - Xineo XIL (XML Import Langage) defines an XML language for transforming various record-based data sources into XML documents, and provides a fully functional XIL processing implementation. This implementation has built-in support for relational (via JDBC) and structured text (like CSV) sources, and is extensible thanks to its public API, allowing dynamic integration of new data source implementations. It also provides an abstraction over output format, and the Xineo implementation can generate output documents into stream or as DOM document. Xineo's implementation built-in data sources include : Relational data via JDBC and Structured text via regular expressions.

CloverETL - CloverETL Features include internally represents all characters as 16bit, converts from most common character sets (ASCII, UTF-8, ISO-8859-1,ISO-8859-2, etc), works with delimited or fix-length data records, data records (fields) are internally handled as a variable-length data structures, fields can have default values, handles NULL values, cooperates with any database with JDBC driver, transforming of the data is performed by independent components, each running as an independent thread, framework implements so called pipeline-parallelism, metadata describing structure of data files (records) can be read from XML and transformation graphs can be read from XML

BabelDoc - BabelDoc is a Java framework for processing documents in linear stages, it tracks documents and can reintroduce documents back into into the pipelines, it is monitorable and configurable through a number of interfaces, it can be run standalone, in server processes or in application servers, it can be reconfigured dynamically by text files and database tables.

Joost - Java implementation of the Streaming Transformations for XML (STX) language. Streaming Transformations for XML (STX) is a one-pass transformation language for XML documents. STX is intended as a high-speed, low memory consumption alternative to XSLT. Since it does not require the construction of an in-memory tree, it is suitable for use in resource constrained scenarios.

CB2XML - CB2XML (CopyBook to XML) is a COBOL CopyBook to XML converter written in Java and based on the SableCC parser generator. This project includes utilities to convert an XML instance file into its COBOL copybook equivalent string buffer and vice versa. You can find additional information about supporting Jurasic systems here.

mec-eagle - JAVA XML XSL B2B integration software:SWING based GUI,an EDI to XML, XML to XML and XML to EDI converter,client-server architecture.All EDI standards are supported:EDIFACT,ANSI X.12,SAP IDOC,XCBL,RosettaNet,Biztalk.Included comm:SMTP,FTP,HTTP(S),PGP/MIME

Transmorpher - Transmorpher is an environment for processing generic transformations on XML documents. It aims at complementing XSLT in order to:

describe easily simple transformations (removing elements, replacing tag and attribute names, concatenating documents...);

allowing regular expression transformations on the content;

composing transformations by linking their (multiple) output to input;

iterating transformations, sometimes until saturation (closure operation);

integrating external transformations.

XPipe - XPipe is an approach to manageable, scaleable, robust XML processing based on the assembly line principle, common in many areas of manufacturing. XPipe as being an attempt to take what was great about the original Unix pipe idea and apply it for structured information streams based on XML.

DataSift - DataSift is a powerful java data validation and transformation framework, aimed at enterprise software development, which provides developers with an extensible architecture they can fully adapt. Almost every feature in it can be configured and extended in some way.

Xephyrus Flume - Flume is a component pipeline engine. It allows you to chain together multiple workers into a pipeline mechanism. The intention of Flume is that each of the workers would provide access to a different type of technology. For example, a pipeline could consist of a Jython script worker followed by a BeanShell script worker followed by an XSLT worker.

Smallx - Smallx supports streaming of XML infosets to allow processing of very large documents (500MB-1GB). Processing is specified in an XML syntax that describes an XML pipeline--which is a sequence of components that consume and produce infosets. This allows chaining of XML component standards like XSLT. Also, there is a full component API that allows developers to easily write their own components.

Nux - Nux is a toolkit making efficient and powerful XML processing easy. It is geared towards embedded use in high-throughput XML messaging middleware such as large-scale Peer-to-Peer infrastructures, message queues, publish-subscribe and matchmaking systems for Blogs/newsfeeds, text chat, data acquisition and distribution systems, application level routers, firewalls, classifiers, etc. Nux reliably processes whatever data fits into main memory (even, say, 250 MB messages), but it is not an XML database system, and does not attempt to be one. Nux integrates best-of-breed components, containing extensions of the XOM, Saxon and Lucene open-source libraries.

KETL - KETL is an extract, transform, and load (ETL) tool designed by Kinetic Networks. KETL includes job scheduling and alerting capabilities. The KETL Server is a Java-based data integration platform consisting of a multi-threaded server that manages various job executors. Jobs are defined using an XML definition language.

Kettle - K.E.T.T.L.E (Kettle ETTL Environment) is a meta-data driven ETTL tool. (ETTL: Extraction, Transformation, Transportation & Loading). No code has to be written to perform complex data transformations. Environment means that it is possible to create plugins to do custom transformations or access propriatary data sources. Kettle supports most databases on the market and has native support for slowly chaning dimensions on most platforms. The complete Kettle source code is over 160,000 lines of java code.

Netflux - Metadata based tool to allow for easier manipulations. Spring based configuration, BSF based scripting support, pluggable JDBC based data sources and sinks. A server and a GUI are planned.

OpenDigger - OpenDigger is a java based compiler for the xETL language. xETL is a language specifically projected to read, manipulate and write data in any format and database. With OpenDigger/XETL you can build Extraction-Transformation-Loading (ETL) programs virtually from and to any database platform.

ServingXML - ServingXML is a markup language for expressing XML pipelines, and an extensible Java framework for defining the elements of the language. It defines a vocabulary for expressing flat-XML, XML-flat, flat-flat, and XML-XML transformations in pipelines. ServingXML supports reading content as XML files, flat files, SQL queries or dynamically generated SAX events, transforming it with XSLT stylesheets and custom SAX filters, and writing it as XML, HTML, PDF or mail attachments. ServingXML is suited for converting flat file or database records to XML, with its support for namespaces, variant record types, multi-valued fields, segments and repeating groups, hierarchical grouping of records, and record-by-record validation with XML Schema.

Talend - Talend Open Studio is full-featured Data Integration OpenSource solution (ETL). Its graphical user interface, based on Eclipse Rich Client Platform (RCP) includes numerous components for business process modelling, as well as technical implementations of extracting, transformation and mapping of data flows. Data related script and underlying programs are generated in Perl and Java code.

Scriptella - Scriptella is an ETL and script execution tool. Its primary focus is simplicity. It doesn't require the user to learn another complex XML-based language to use it, but allows the use of SQL or another scripting language suitable for the data source to perform required transformations.

ETL Integrator - ETL (a highly unimaginative name) consists of 3 components. An ETL service engine that is a JBI compliant service engine implementation which can be deployed in a JBI container. An ETL Editor that is a design time netbeans module which allow users to design ETL process in a graphical way. An ETL Project that is a design time netbeans module which allows users to package ETL related artifacts in a jar file which could be deployed onto the ETL service engine.

Jitterbit - Jitterbit can act as a powerful ETL tool. Operations are defined, configured, and monitored with a GUI. The GUI can create document definitions, from simple flat file structures to complex hierarchic files structures. Jitterbit includes drag-and-drop mapping tool to transform data between your various system interfaces. Furthermore, one can set schedules, create success and failure events and track the results for your integration operations. Jitterbit supports Web Services, XML Files, HTTP/S, FTP, ODBC, Flat and Hierarchic file structures and file shares.

Apatar - Apatar integrates databases, files and applications. Apatar includes a visual job designer for defining mapping, joins, filtering, data validation and schedules. Connectors include MySQL, PostgreSQL, Oracle, MS SQL, Sybase, FTP, HTTP, SalesForce.com, SugarCRM, Compiere ERP, Goldmine CRM, XML, flat files, Webdav, Buzzsaw, LDAP, Amazon and Flickr. No coding is required to accomplish even a complex integration. All metadata is stored in XML.

Spring Batch - Spring Batch is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications. Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management. It also provides more advance technical services and features that will enable extremely high-volume and high performance batch jobs though optimization and partitioning techniques.

JasperETL - JasperETL was developed through a technology partnership with Talend. JasperETL includes Eclipse based user interfaces for process design, transformation mapping, debugging, process viewing. The project includes over 30 connectors like flat files, xml, databases, email, ftp and more. It includes wizards to help configure the processing of complex file formats including positional, delimited, CSV, RegExp, XML, and LDIF formatted data.

Pentaho Data Integration - Pentaho Data Integration provides a declarative approach to ETL where you specify what to do rather than how to do it. It includes a transformation library with over 70 mapping objects. In includes data warehousing capability for slowly changing and junk Dimensions. Includes support for multiple data sources including over 25 open source and proprietary database platforms, flat files, Excel documents, and more. The architecture is extensible with a plug-in mehcanism.

Mural - Mural is an open source community with the purpose of developing an ecosystem of products that solve the problems in Master Data Management (MDM). Projects include: Master Index Studio which provides the supports the creation of a master index through the matching, de-duplication, merging, and cleansing . Data Integrator which provides extract, transform, load capability and a wide variety of data formats. Data Quality which features matching, standardization, profiling,and cleansing capabilities. Data Mashup Data Mashup which provides data mashup capability. Data Migrator which supports the migration of database objects across database instances

Smooks - Smooks provides a wide range of Data Transforms. Supports many different Source and Result types - XML/CSV/EDI/Java/JSON to XML/CSV/EDI/Java/JSON. It supports binding of Java Object Models from any data source. It is designed to process huge messages in the GByte range.

Data Pipeline - Data Pipeline provides data conversion, data processing, and data transformation. The toolkit has readers and writers for common file formats (CSV, Excel, Fixed-width, JDBC) along with decorators that can be chained together to process and transform data (filter, remove duplicates, lookups, validation).

Ubuntu9.04安装配置

Sat, 30 May 2009 03:19:18 GMT

下面描述的是Ubuntu9.04服务器版安装配置：
1. 网络设置：

编辑 interfaces文件。

$ sudo vi /etc/network/interfaces

eth0配置如下：

auto eth0

address 192.168.1.123

netmask 255.255.255.0

gateway 192.168.1.1

保存退出后，使用重启networking命令让新配置生效。

$ sudo /etc/init.d/networking restart

也可以通过如下命令重启网卡，让新配置生效,好处是不影响其他网络接口。

$ sudo ifdown eth0

$ sudo ifup eth0

如果只是要临时改变IP地址，则不用修改interface.只用ifconfig使用即可，不过当系统重启动后，系统后会恢复interfaces中的配置上。

$ sudo ifconfig eth0 192.168.1.111 netmask 255.255.255.0

设置DNS：

$ sudo vi /etc/resolv.conf

nameserver 61.235.70.252

nameserver 211.98.4.1

2. 更新源：

$ sudo apt-get update

中文操作系统环境和中文输入法，操作系统在连上互联网后会自动提示安装中文环境和中文输入法。

3. 安装telnet

1. sudo apt-get install xinetd telnetd

2. 安装成功后，系统也会有相应提示（好象7.10才有，6.10就没看到）

　　sudo vi /etc/inetd.conf并加入以下一行

　　telnet stream tcp nowait telnetd /usr/sbin/tcpd /usr/sbin/in.telnetd

3. sudo vi /etc/xinetd.conf并加入以下内容：

# Simple configuration file for xinetd
#
# Some defaults, and include /etc/xinetd.d/
defaults
{
# Please note that you need a log_type line to be able to use log_on_success
# and log_on_failure. The default is the following :
# log_type = SYSLOG daemon info
instances = 60
log_type = SYSLOG authpriv
log_on_success = HOST PID
log_on_failure = HOST
cps = 25 30
}

　　includedir /etc/xinetd.d

4. sudo vi /etc/xinetd.d/telnet并加入以下内容：

# default: on
# description: The telnet server serves telnet sessions; it uses
# unencrypted username/password pairs for authentication.
service telnet
{
disable = no
flags = REUSE
socket_type = stream
wait = no
user = root
server = /usr/sbin/in.telnetd
log_on_failure += USERID
}

5. 重启机器或重启网络服务sudo /etc/init.d/xinetd restart

4. 设置语言环境：

/etc/environment文件如下：

代码:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/games"
LANG="zh_CN.GBK"
LANGUAGE="zh_CN:zh:en_US:en"
LC_CTYPE=zh_CN.GBK
LC_ALL=zh_CN.GBK
GST_ID3_TAG_ENCODING=GBK
ID3_TAG_ENCODING=GBK

#sudo vi /etc/profile 文件
在最后加入一句
export LANG=en_US

5. 安装vsftpd

输入：

sudo apt-get install vsftpd

如果没换源可能会提示你使用光盘，插入光盘

sudo cp /etc/vsftpd.conf /etc/vsftpd.conf.old

然后可以改动了：

sudo vi /etc/vsftpd.conf

#不让匿名用户使用

#anonymous_enable=YES

#本地用户可用

local_enable=YES

#可用写操作

write_enable=YES

#不需要显示某目录下文件信息

#dirmessage_enable=YES

#加点banner提示

ftpd_banner=Hello~~

#FTP服务器最大承载用户

max_clients=100

启动ftp服务

#sudo service vsftpd start

安装zip：

#sudo apt-get install zip unzip

6. 设置Java环境

方法有三：
一、临时设置
export JAVA_HOME= /home/liupinghua/jdk1.5.0_18

二、当前用户的全局设置
打开~/.bashrc，添加行：
export JAVA_HOME= /home/liupinghua/jdk1.5.0_18
注销
这样每次以此用户登录Ubuntu，该环境变量都会生效。

三、所有用户的全局设置
$ vi /etc/profile
在里面加入：
export JAVA_HOME= /home/liupinghua/jdk1.5.0_18
注销
这样不管是以哪个用户登录，该环境变量都会生效。

7. 启动Tomcat

#sudo ./startup.sh

l 设置开机自动启动Tomcat

1）、使用tomcat自带的jsvc工具，生成脚本使tomcat自动启动

cd tomcat/bin
tar -zxvf jsvc.tar.gz
cd jsvc-src
chmod +x configure
./configure --with-java=$JAVA_HOME
make
cd native
gedit Tomcat5.sh
--根据需要修改下面文件的参数

# Adapt the following lines to your configuration
JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun
CATALINA_HOME=/home/user/tomcat/
DAEMON_HOME=/home/user/tomcat/
TOMCAT_USER=user
#为tomcat设置一个启动用户，非root用户
#出于安全性考虑，如果是root用户，jsp执行权限太高，有被注入的问题

# for multi instances adapt those lines.
TMP_DIR=/var/tmp
PID_FILE=/var/run/jsvc.pid
CATALINA_BASE=/home/user/tomcat/

#CATALINA_OPTS="-Djava.library.path=/home/jfclere/jakarta-tomcat-connectors/jni/native/.libs"
CLASSPATH=
$JAVA_HOME/lib/tools.jar:
$CATALINA_HOME/bin/commons-daemon.jar:
$CATALINA_HOME/bin/bootstrap.jar

case "$1" in
  start)
    #
    # Start Tomcat
    #
    $DAEMON_HOME/bin/jsvc-src/jsvc
    -user $TOMCAT_USER
    -home $JAVA_HOME
    -Dcatalina.home=$CATALINA_HOME
    -Dcatalina.base=$CATALINA_BASE
    -Djava.io.tmpdir=$TMP_DIR
    -wait 10
    -outfile $CATALINA_HOME/logs/catalina.out
    -errfile '&1'
    $CATALINA_OPTS
    -cp $CLASSPATH
    org.apache.catalina.startup.Bootstrap
    #
    # To get a verbose JVM
    #-verbose
    # To get a debug of jsvc.
    #-debug
    exit $?
    ;;

  stop)
    #
    # Stop Tomcat
    #
    $DAEMON_HOME/bin/jsvc-src/jsvc
    -stop
    org.apache.catalina.startup.Bootstrap
    exit $?
    ;;

  *)
    echo "Usage tomcat.sh start/stop"
    exit 1;;
esac

将修改的文件复制到/etc/init.d/中
cp Tomcat5.sh /etc/init.d/tomcat.sh
修改执行权限
sudo chmod +x tomcat.sh
这样tomcat就会随着系统自动启用
测试：
sudo /etc/init.d/tomcat.sh start
sudo /etc/init.d/tomcat.sh stop

8. 加入第二块硬盘

　　给硬盘分区

　　在slackware下有两个分区软件fdisk和cfdisk

例如我们已经有一个硬盘了，现在添加另一个硬盘到系统

那么我们根据命名规则知道这个新添加的硬盘应该是hdb。我们用下面命令给硬盘分区

fdisk /dev/hdb

　　你也可以用cfdisk来分区，命令如下

cfdisk /dev/hdb

　　格式化硬盘

　　格式化成ext3格式

mkfs.ext3 /dev/hdb1

　　格式化成reiserfs的格式

mkfs.reiserfs /dev/hdb1

　　让硬盘启动自动挂载

　　例如挂载/dev/hdb1分区到/mnt/hd目录下

用vi编辑/etc/fstab文件，加入如下内容

/dev/dhb1 /mnt/hd reiserfs defaults 1 1

9. 中文环境问题

在/etc/environment这个文件里面可以设置全局的LANG变量

$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:
/usr/games"
LANG="zh_CN.UTF-8"
LANGUAGE="zh_CN:zh:en_US:en"

但是当我们sudo -i进root用户时, LANG又变成了C

# locale

LANG=C

LANGUAGE=C

LC_CTYPE="C"

LC_NUMERIC="C"

LC_TIME="C"

LC_COLLATE="C"

LC_MONETARY="C"

LC_MESSAGES="C"

LC_PAPER="C"

LC_NAME="C"

LC_ADDRESS="C"

LC_TELEPHONE="C"

LC_MEASUREMENT="C"

LC_IDENTIFICATION="C"

LC_ALL=

这是root的~/.profile里面的设置造成:

~# cat .profile

# Installed by Debian Installer:

# no localization for root because zh_CN.UTF-8

# cannot be properly displayed at the Linux console

LANG=C

LANGUAGE=C

因为有些情况下显示有问题, 所有root中强制设置成了LANG=C.