marksway's Blog

[광고] 해커스

2021-01-08T05:00:00.002-08:00

블로그 본문

광고배너

공정위 문구

[IT] 데이터 레이크 Data Lake

2020-12-10T14:57:00.003-08:00

[IT 시각]

데이터 레이크는 대규모의 다양한 원시 데이터 세트를 기본 형식으로 저장합니다.

(원시 데이터는 특정 목적을 위해 처리되지 않은 데이터를 말합니다.)

(레이크라고 부르는 이유는 필터링되거나 패키지화되지 않은 수역과 같은 자연 상태의 데이터 풀을 저장하기 때문입니다.)

데이터는 여러 소스에서 레이크로 흐르며 원래 형식으로 저장됩니다.

데이터 레이크를 사용하면 모든 데이터가 보존됩니다.

데이터 레이크를 사용하면 정제되지 않은 데이터를 볼 수 있습니다.

데이터는 얼마 후 바로 분석에 사용될 수도 있고 전혀 사용되지 않을 수도 있습니다.

데이터 레이크에 있는 데이터는 쿼리되기 전까지는 정의되지 않습니다.

데이터 레이크에 있는 데이터는 분석을 위해 필요할 때 변환됩니다.

데이터 사이언티스트들은 보다 고도화된 분석 툴이나 예측 모델링을 사용하여 원시 데이터에 액세스할 수 있습니다.

[비지니스 시각]

> 무한한 데이터에서 가치를 창출한다.

모든 유형의 정형 데이터와 다양한 비정형 데이터를 저장할 수 있다.(IoT 데이터, 소셜 미디어 게시물 등)

> 미리 답을 예측할 필요가 없다.

원시 데이터를 저장하기만 하면 된다. 원시 데이터에 대한 이해와 인사이트가 향상될 때 데이터를 정제하면 된다.

> 인사이트를 얻을 수 있다.

데이터 인사이트를 얻을 수 있는 다양한 방식을 활용할 수 있다.

[데이터 레이크와 테이터 웨어하우스의 유사점, 차이점]

데이터 레이크와 데이터 웨어 하우스는 종종 혼동되지만, 이 둘은 동일하지 않으며 그 목적도 다릅니다.

>유사점

둘 다 빅데이터를 위한 데이터 저장소

>차이점

데이터 웨어하우스는 보고를 위해 설계된 구조화된 데이터 모델을 제공

데이터 레이크는 현재 정의된 목적이 없는 비정형 원시 데이터를 저장

ABPP 0022 Deployment

2011-01-07T07:42:00.000-08:00

1. Select "Prepare For Deplyment"

2. Enter locations

3. Making deployment file

4. You can distribute "deploy.jar" to customer

ABPP 0017 User Authorization

2011-01-07T07:30:00.002-08:00

1. Creaate new user activity file

2. Create new user activity, ex) seller_activity

3. Post "updateActivityInDB" method.

4. Map "role" and "activity" using "ROLE_TEMPLATE" table.

5. You can control authority for each workflow.

6. You can control authority for each screen component.

ABPP 0016 User Authentication

2011-01-07T07:30:00.000-08:00

1. Import archived service "USER_SECURITY"

2. Select "Archived Service"

3. Select "USER_SECURITY"

4. Generate incremental schema by solution setup.

5. Create user profile data in ../data/user_data.xml

<REQUESTS ServiceName="USER_SECURITY">
<REQUEST Name="addRole">
  <ID Value="BUYER"/>
  <NAME Value="BUYER"/>
</REQUEST>
<REQUEST Name="addRole">
  <ID Value="SELLER"/>
  <NAME Value="SELLER"/>
</REQUEST>

<REQUEST Name="addUserGroup">
  <ID Value="BOOKSTORE_BUYER_UGP"/>
  <GRP_NAME Value="BOOKSTORE_BUYER"/>
  <ORG_ID Value="ORG_1"/>
  <GRP_TYPE Value="ENTERPRISE"/>
  <DESC Value="Bookstore Buyer User Group"/>
  <AUTO_ACTIVATE Value="true"/>
  <ASSIGN_ROLE_TEMPLATES>
   <ROLE_TEMPLATE Value="BUYER"/>
  </ASSIGN_ROLE_TEMPLATES>
</REQUEST>
  <REQUEST Name="addUserGroup">
  <ID Value="BOOKSTORE_SELLER_UGP"/>
  <GRP_NAME Value="BOOKSTORE_SELLER"/>
  <ORG_ID Value="ORG_1"/>
  <GRP_TYPE Value="ENTERPRISE"/>
  <DESC Value="Bookstore Seller User Group"/>
  <AUTO_ACTIVATE Value="true"/>
  <ASSIGN_ROLE_TEMPLATES>
   <ROLE_TEMPLATE Value="SELLER"/>
  </ASSIGN_ROLE_TEMPLATES>
</REQUEST>

<ADD_DOCUMENT Name="USER_PROFILE">
  <ID Value="buyer1"/>
  <LOGIN_NAME Value="buyer1"/>
  <PASSWORD Value="pw"/>
  <ORG_ID Value="ORG_1"/>
  <CONTACT_ID Value="ct-1"/>
  <LOCALE Value="en_US"/>
  <USER_GRP_ID Value="BOOKSTORE_BUYER_UGP"/>
</ADD_DOCUMENT>
<ADD_DOCUMENT Name="USER_PROFILE">
  <ID Value="seller1"/>
  <LOGIN_NAME Value="seller1"/>
  <PASSWORD Value="pw"/>
  <ORG_ID Value="ORG_1"/>
  <CONTACT_ID Value="ct-1"/>
  <LOCALE Value="en_US"/>
  <USER_GRP_ID Value="BOOKSTORE_SELLER_UGP"/>
</ADD_DOCUMENT>

</REQUESTS>

6. Load user profile data

7. edit xserver.xml

8. "User Authentication" is on.

ABPP 0014 creating simple UI node for "creating new order"

2011-01-05T21:43:00.000-08:00

view.x2ps vs start.x2ps

2011-01-05T02:07:00.001-08:00

There are two reserved actions with respect to workflow interaction.
- view.x2ps: This is for invoking a continue workflow action wherein the
control goes to the post processing section of the current UI node. In our case
we are specifying view.x2ps so that our workflow would continue from this
UI node to go to the' add book from catalog' screen; control will then return to
this UI node for further actions.
- start.x2ps: This is for starting a new workflow. This needs two extra
parameters, workflow name and the service name which holds that workflow

ABPP 0013 executing debugging

2011-01-04T18:03:00.000-08:00

1. Click set debugger.

2. Set debug breaking point.

3.When you meet breaking point in your browser, debuggin window will pop up authmatically.

4. You cau see variable's value.

ABPP 0012 making simple UI with table editor

2011-01-04T17:48:00.000-08:00

1. add a new workflow "wf_select_book.xml"

2. Edti new table editor's properties
name : te_1
description : te_1
service : SVC_STORE
table name : BOOK_MASTER
Model : SELECT
select property : BOOK_ID
is multi selected : Yes
assign selecttion : xml_book_selected

[Edit Next Nodes Tab]
output var : xml_output
output type : xml

3. Edit done properties.
name : dn_1
node input : xml_book_selected
type : xml
action : ...

4. Right-click in white background, and "click add to navigation"

5. You can see new workflow in menu.

ABPP 0011 unit testing

2011-01-04T08:16:00.000-08:00

1. Edit xserver.xml.

2. After running server, you can see error message.

3. For unit testing,
    Click "Post" icon.
    ->Select a service.
    ->Enter input xml.

4. You can see output result.

ABPP 0010 explanation about X-Rule

2011-01-03T13:33:00.000-08:00

In ABPP,
a rule file have many methods.
a method is composed of 3 parts(Input/ Output/ Action).

Input/Output must be XML format.
. thisParam: This variable is a reference to the input xml
. thisReturn: This variable is a reference to the output xml

ABPP 0009 making new rules

2011-01-03T12:18:00.000-08:00

Normally in java, we have a java file which is composed of class/member.
In ABPP, we can map like this.
a java file -> a rule file
a class/member -> a method

And ABPP's input/output must be XML format.

1. Add a rule to a service.

2. Add a rule file to a rule.

3. Enter a rule file name.

4. You can edit a rule file.
A rule will be seperated by 3 groups(input/ output/ action).

ru_order.xml

ABPP 0008 creating workflow

2011-01-03T11:04:00.000-08:00

1. Insert workflow file.

2. Enter workflow name.

3. Add "Start", "Table Editor", "Done"

4. Add values to "Table Editor"

5. Map menu and workflow.

6. Restart ABPP Server.

ABPP 0007 creating physical tables

2011-01-03T10:32:00.000-08:00

1. By "Solution Setup", you can create physical tables.

2. Select proper services.

3. Select full or incremental generation.

ABPP 0006 creating service

2011-01-03T09:53:00.002-08:00

ABPP project will be managed by each services.
So, each business process will be developed on each service.

1. Create Service

2. Enter Service Name.

3. Select Base Model Set.

4. "SVC_STORE" is created.

5. In your folder, you can see files in "properties" and new folder in "xservice".

ABPP 0005 creating table

2011-01-03T07:03:00.000-08:00

In ABPP, a physical table will be created by a MTT file.

Here is a sample MTT file including 1 primary key.

BOOK_MASTER.mtt

To remember,
1. file name is physical table name.
2. property is physical column name.
3. String_xx, Integer, Float_xx_xx, Date
4. primary key will be managed specially.

> <document>
>   <facets>
>     <documentFacets>
>       <documentFacet Type="miscFacet" Active="true" />
>     </documentFacets>
>   </facets>
>   <documentation>Table containing all books in my store.</documentation>
>   <properties>
>     <property Name="BOOK_ID" PhysicalName="BOOK_ID" Active="false" Required="true" Type="String_100">
>       <constraints Caption="constraints" />
>     </property>
>     <property Name="BOOK_NAME" PhysicalName="BOOK_NAME" Active="false" Type="String_100" />>     <property Name="BOOK_AUTHOR" PhysicalName="BOOK_AUTHOR" Active="false" Type="String_100" />
>     <property Name="BOOK_PRICE" PhysicalName="BOOK_PRICE" Active="false" Type="Float_7_2" />
>     <property Name="BOOK_RELEASE_DATE" PhysicalName="BOOK_RELEASE_DATE" Active="false" Type="Date" />
>     <property Name="BOOK_URL" PhysicalName="BOOK_URL" Active="false" Type="String_100" />
>   </properties>
>   <keys>
>     <key Active="true" Name="PK_BOOK_MASTER" PhysicalName="PK_BOOK_MASTER" Kind="PRIMARY">
>       <property Name="BOOK_ID" />>     </key>
>   </keys>
>   <links />
> </document>

ABPP 0004 changing menu

2011-01-03T03:00:00.000-08:00

1. Change solution's displayText.

2. Right-click pad item.

3. You can see a new pad-item.

ABPP 0003 see it in explore browser

2011-01-03T02:39:00.000-08:00

1. Start your weblogic domain.

2. Your weblogic started.

3. Start your abpp server in your studio.

4. Enter URL : http://yourcomputer:weblogicport/base/core/login.jsp

5. blank window is opened.

ABPP 0002 start ABPP server

2011-01-03T02:31:00.000-08:00

1. Click start button.

2. Select all services.

3. You can see logs.

> java version "1.5.0_14"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
> Java HotSpot(TM) Client VM (build 1.5.0_14-b03, mixed mode, sharing)
> abpp.properties=D:\i2kjs\abpp\6.3.1\abpp.properties
> Comparing system IP 10.152.49.30 or MAC Address 00-1C-25-1E-EB-F8 with license IP 00-1C-25-1E-EB-F8
> ABPP Server 6.3.1
> Build Number 0484, Build Date Jun-19-2009 07:35 PM, Vendor i2 Technologies, Inc.
>
> Loading cfg/properties/boot_services.xml
> NEW UPDATE FORM
> ServerSetup.timingStats = false
> Initializing thread pool with create increment:3 and max idle limit:12
> Default threshold=15
> Starting server with Worker Threads=5 AsynchThreads=5
> Set threshold=10
> xserver :LogLevels [ All, Audit, BenignException, Debug, EnterTrace, Error, Exception, ExitTrace, Info, InterServiceCallTrace, IntraServiceCallTrace, SqlCallTrace, SystemInfo, Warning, XPathFnTrace ]
> Creating initial connections :5
> Loading database meta info from:com/i2/xcore/dblib/oracle.xml
> PosterSetup.timingStats = false
> Turning on API signature validation for service : PGL
> PGL :LogLevels [ Error, Exception ]
> Turning on API signature validation for service : PROJECT001UI
> PROJECT001UI :LogLevels [ All, Audit, BenignException, Debug, EnterTrace, Error, Exception, ExitTrace, Info, InterServiceCallTrace, IntraServiceCallTrace, SqlCallTrace, SystemInfo, Warning, XPathFnTrace ]
> **   1 07:16:07¿??? GET_DOCUMENT         CRYPTION_KEY                   None
> **   2 07:16:07¿??? ADD_DOCUMENT         CRYPTION_KEY                   None
> **   3 07:16:07¿??? ENABLE_DEBUG         -                              None
> Enabling debugger
> Binding debugger: //localhost:1644/debugger-xcore-4444
> Accepting connections on port ** 4444

ABPP 0001 making new project

2011-01-03T01:32:00.000-08:00

0001 ABPP : making new project

1. First I created a user in Oracel Database.

create user abpptest001
identified by abpptest001;

grant connect, resource to abpptest001;

grant dba to abpptest001 with admin option;

2. Select "New".

3. Enter "Project Name", "Project Folder".

4.Enter "User Name","User Password","DB Host",'"DB Port","SID".

5.Check UI Workflows.

6.You can see in Navigator.

7.In you folder, you can see the followings.

This blog introduces Korea's SCM information.

2011-01-02T21:02:00.000-08:00

This blog introduces Korea's SCM information.