The Home of PuppetAR Master

Into RealityKit Coding AR using XCode

2019-08-14T19:52:00.001-07:00

I am slowly trying to push myself to code using XCode and Swift, to make a more sophisticated AR experience. Apple provides us with Reality Composer app that comes officially with iOS 13, iPadOS, and also MacOS Catalina. And this is already quite powerful in its own way.

Further more RealityKit framework is also introduced and supposedly can make it easier for AR App development.

Imagine the possibilities:
You could have an App with multiple RCProjects, which contains multiple scenes, each scene is anchored differently, and each scene contains objects. On top of that, if you make it into an app, you can have an interactive story telling animation or games.

One of the best example AR is Apple's own AR Memoji / Animoji app that is bundled into iMessage. That is quite sophisticated example of AR although it is mostly for Face AR.

Although to get into that level 9999 we probably need to start simple!

EASY ANCHORING

Making a 3D Augmented Reality environment and experience has indeed become so much easier, first of all with: Easy ANCHOR system all across Apple ARKit and RealityKit. To place 3D scene or objects we simply choose which anchor we like to place our AR into: Floor (horizontal), Wall (vertical), Image, Face, Object, etc.

AR QUICK LOOK & REALITY COMPOSER

On top of that, last year with iOS 12 and introduction of USDZ and AR Quick Look, this in itself also helping 3D artists and designers and also general users to get used to the idea of general usage AR objects (USDZ).

With Reality Composer, any 3D artists with access to MacOS can easily prepare their 3D assets and create AR Composition using the familiar environment. Adding simple behaviour interaction and animation also become amazingly simple. And you can always export and share result as .reality format that can be viewed via AR Quick Look of iOS 13 and iPadOS.

Enter REALITY KIT and ARKIT

A slightly more complex yet powerful way to make AR experience is packaging it an App, which you create using XCode, while still utilizing the Reality Composer => .rcproject which you can load and develop with all the already available UI creation and game logics.

I am not a full developer, but I slowly wanted to familiarise myself with the Swift coding environment, which hopefully be beneficial for future AR experience making.

Let's get started!

First of all, XCode 11 does provide us with a template for AR that utilizes RealityKit:

This is a great place to start. You can see how it loads the AR Scene from "Experience.rcproject" file that you can Open in Reality Composer.

You see the reason you can .loadBox() is that because the Experience.rcproject can have multiple SCENES, and this particular project only have 1 scene named "Box".

Now, if you quickly open the Experience.rcproject inside Reality Composer and adding some new objects into the scene and NAME it, you can then modify this new ENTITY inside XCode.

I am adding this pink colored plastic "spiral" object.

        // Load the "Box" scene from the "Experience" Reality File

        let boxAnchor = try! Experience.loadBox()

        let spiralEntity = boxAnchor.spiral

        spiralEntity?.setPosition(SIMD3<Float>(0.0, 0.1, 0.0), relativeTo: boxAnchor)

Let's supposed we also add code above, which basically load our Spiral object and then set its position to 0.0, 0.1, 0.0 XYZ (Y is up and down) and relative to its boxAnchor (root), we will then get this:

SIMD thing is surprised me at first, however I realized every value needs to be a certain type.

        suzanneTemplate.scale = SIMD3(repeating: Float(0.5))

You see the Spiral object is moved into location that is relative to the anchor we specified.

ADDING CUSTOM MESH PRIMITIVES INTO SCENE

A more challenging step is when we started to code and adding our own 3D objects using Swift code like below...

You can quickly make basic Materials or even sophisticated one like OcclusionMaterial this way:

        // Making a couple of custom box mesh

        let boxMesh = MeshResource.generateBox(width: 0.1, height: 0.1, depth: 0.1) // 0.2 is in meter and quite large

        let material = SimpleMaterial(color: UIColor.red, roughness: 0.3, isMetallic: true)

        let boxOne = ModelEntity(mesh: boxMesh, materials: [material])

        //let boxMesh = MeshResource.generateBox(width: 0.2, height: 0.2, depth: 0.2)

        let materialTwo = SimpleMaterial(color: UIColor.blue, roughness: 0.3, isMetallic: true)

        let boxTwo = ModelEntity(mesh: boxMesh, materials: [materialTwo])

        let materialThree = SimpleMaterial(color: UIColor.green, roughness: 0.3, isMetallic: true)

        let boxThree = ModelEntity(mesh: boxMesh, materials: [materialThree])

        //let monkey = try! Suzanne.SceneMonkey.load(named: "Monkey")

        //monkey.addChild(occlusionBox)

        let sphereMesh = MeshResource.generateSphere(radius: 0.1)

        let materialFour = SimpleMaterial(color: UIColor.purple, roughness: 0.3, isMetallic: true)

        let boxFour = ModelEntity(mesh: sphereMesh, materials: [materialFour])

        let materialFive = SimpleMaterial(color: UIColor.orange, roughness: 0.3, isMetallic: true)

        let boxFive = ModelEntity(mesh: sphereMesh, materials: [materialFive])

        // Prepare the parents masters

        let floor = monkeyAnchor.floor

        let monkeyMaster = monkeyAnchor.monkey

        let monkeyTwo = monkeyAnchor.monkeytwo

        //let monkeyThree = monkeyAnchor.monkeythree

        // Parents created objects

        floor!.addChild(boxOne)

        floor!.addChild(boxTwo)

        floor!.addChild(boxThree)

        monkeyMaster!.addChild(boxFour)

        monkeyTwo!.addChild(boxFive)

        monkeyTwo!.addChild(myAnchor)

        boxTwo.setPosition(SIMD3<Float>(0.5, -0.0, -0.0), relativeTo: floor) // Y is up and down, X is L and R

        boxThree.setPosition(SIMD3<Float>(-0.5, -0.0, -0.0), relativeTo: floor) // Y is up and down, X is L and R

        //boxFour.position.y = 0.05

        //boxFive.position.y = 0.02

        boxFour.setPosition(SIMD3<Float>(-0.0, 2.0, -0.0), relativeTo: monkeyMaster) // Y is up and down, X is L and R

        boxFive.setPosition(SIMD3<Float>(0.5, 1.0, -0.0), relativeTo: monkeyTwo) // Y is up and down, X is L and R

In a nutshell, what I am doing above is:

- Create Mesh data using MeshResource

- Create new Material

- Assign Material into Mesh and turn it into ENTITY

- Parent the new Mesh into our existing object or anchor

- Setting its position relative to another object

Let me modify the code a little bit since the one above was taken from another project, but I want to use it in our project which only contains Box (Steel Box) and Spiral object.

        // Making a couple of custom box mesh

        let boxMesh = MeshResource.generateBox(width: 0.03, height: 0.03, depth: 0.03) // 0.2 is in meter and quite large

        let material = SimpleMaterial(color: UIColor.red, roughness: 0.3, isMetallic: true)

        let boxOne = ModelEntity(mesh: boxMesh, materials: [material])

        //let boxMesh = MeshResource.generateBox(width: 0.2, height: 0.2, depth: 0.2)

        let materialTwo = SimpleMaterial(color: UIColor.blue, roughness: 0.3, isMetallic: true)

        let boxTwo = ModelEntity(mesh: boxMesh, materials: [materialTwo])

        // Prepare the parents masters

        let boxEntity = boxAnchor.steelBox

        //let monkeyThree = monkeyAnchor.monkeythree

        // Parents created objects

        boxEntity?.addChild(boxOne)

        spiralEntity!.addChild(boxTwo)

        boxOne.setPosition(SIMD3<Float>(0.0, 0.2, 0.0), relativeTo: spiralEntity) // Y is up and down, X is L and R

        boxTwo.setPosition(SIMD3<Float>(-0.0, 0.2, 0.0), relativeTo: boxEntity) // Y is up and down, X is L and R

You can see there are 2 new floating random box with BLUE and RED material, parented into our Box and Spiral objects. If you add simple tap interaction using Reality Composer (so much simpler than actually coding it), you will see that the parented Blue and Red Box will follow.

ADDING CUSTOM 3D USDZ

If you want to manually add your own custom USDZ manually, not via .rcproject, you can do that also, something like below: (create anchor, addchild)

        // Want to custom model...

        let myAnchor = AnchorEntity(plane: .horizontal)

        arView.scene.anchors.append(myAnchor)

        let myObject = try! Entity.loadModel(named: "evilmask")

        myAnchor.addChild(myObject)

        myObject.setScale(SIMD3<Float>(repeating: 0.1), relativeTo: myAnchor) // scale own object to 0.1

        myObject.setPosition(SIMD3<Float>(0.0, 0.0, 0.0), relativeTo: myAnchor)

NEXT STEP...

Ok, I guess this is a good starting point to dive into SWIFT and RealityKit to modify .rcproject for AR. You should now be more familiar with Reality Composer role in making AR app and how to modify the existing scene using RealityKit and a bit of coding.

Perhaps the next step is to get used to also dealing with SCREEN TAP INTERACTION. There is a good article on this written by Ash from USDZSHARE:
https://usdzshare.com/using-occlusion-in-reality-kit/

See that you can add fancy shared like "Occlusion".

You can perhaps play with Swift and making a loop that generate 100 entity of array of Boxes into the AR scene.

CHALLENGE:
Let see if you can make a CLONE of ENTITY and then place it randomly in the scene. Maybe that's my next article.

You could get to the point where you can start making GAME with Game Logics, interaction, etc, but that's maybe Level 100!!

The most important is that this process I think help us to think of AR and its usage. Perhaps learning about AR is also to think about CODING?

BSLIVE Remixing Idea and 3D Mesh from Sketchfab using Blender 3D and Reality Composer

2019-07-28T19:54:00.001-07:00

SV Baking Animated Point Cloud With Nodes

2019-07-27T18:32:00.001-07:00

I am proposing a simple way to bake animated point cloud data (PLY) in Blender using Sverchok Nodes that is easy and simple to understand. The resulting bake is a single Alembic / ABC file that is much smaller and easy to load using Blender for further processing.

ARKIT / Apple Reality Composer and RealityKit

2019-07-27T03:41:00.000-07:00

This is a fun quick AR composition using iPad Pro, based on topic "Moon Landing 2019"

At WWDC 2019, Apply announces a couple of new tools to help users to create AR or Augmented Reality experience and contents. They come in forms of Reality Composer app, which will be available with iOS 13 for iPhone and iPad (iPadOS), and also RealityKit inside XCode which will help artists and developers a lot in term of handling "reality" contents and 3D assets. This was built on top of ARKit and also USDZ file format which was proposed a year ago.

I am personally super excited with what Reality Composer (RC) brought into the AR world! There has not been anything quite like RC especially on mobile device such as iPhone and iPad. So far I have been testing RC with iPhone X (A11 chip) and also iPad Pro 2018 (A12 chip). RC app performs amazingly well and really can handle all sorts of USDZ assets.

AR ANCHORS BECOME SO SIMPLE!

When you open the app the first time, whether the version of Reality Composer on MacOS Catalina or iOS 13 or iPadOS, you will get this option menu to choose an Anchor: Horizontal (floor), Vertical (wall), Image, Face, or Object. You might have more in the future.

I cannot be happier with this. In the past, one needs to use XCode and ARKit and some coding to make this happen. Not anymore, for AR Scene now can be built and anchored into real life environment simply by selecting an anchor!

Simple Easy To Use 3D Navigations

When you open RC the first time, even if you are not a 3D artist, for sure you will start to get to know the 3D scene and environment. This is really as simple as it can get. It does not try to be another 3D package, but clearly this app is for composing layout of 3D that you can later use inside XCode to publish as App.

To start with Reality Composer, I recommend you to try and source some USDZ probably from webonline service like USDZShare or maybe from Apple USDZ gallery. Or even better, try and download some 3D assets from Sketchfab, convert them into USDZ using "usdzconvert" app from Apple (comes with USDPython) and bring them all into your device and import the USDZ into Reality Composer app.

You will learn to navigate in 3D and dealing with 3D assets. You can switch to AR Mode (on iOS devices) and quickly transform (move, scale, rotate) 3D objects virtually.

The control and navigation is similar to XCode handling of 3D with ARKit and SceneKit. You can actually use MacBookPro TrackPad:

- Hold and drag = tumble

- Two Finger Pinch In and Out = zoom / dolly

- Two Finger drag = panning

Shin Godzilla model from Sketchfab, converted into USDZ and imported into Reality Composer

BUILD IN PREFABS & LIBRARIES OF 3D OBJECTS

Reality Composer comes with some ready to use 3D Primitives or Geometries: Box, Sphere, Torus, Capsules, etc and plus also some varieties of objects (buildings, sport equipments, foods, frames, etc) given by Apple for users to start composing and testing for AR.

Each 3D Primitives comes with some Properties that you can modify. There are also some preset Materials, but you cannot really add your own Texture yet and still pretty limited. But they are super handy as a prototype materials.

On top of all that, TEXT Object is also provided. So you can easily insert TEXT object as actual interactive objects. This is a lot more efficient than having to bring your own baked text mesh, which usually be heavy in term of polygon counts.

PHYSICS and INTERACTIVITY

Further more, RC also allows you to turn on dynamics and add some physics into the 3D assets. This is pretty amazing. A very obvious to try is to build tower of cans or maybe setting up bowling pins and throwing balls into it. This is super easy to create!

Interactivities also come in form of BEHAVIOURS that you can simply attach into objects so that they would react based on trigger like tap, etc.

SOUND BEHAVIOURS

Just like a movie or animated scene, SOUNDS are also included as part of Reality Composer "sandbox environments".

INTO XCODE WITH REALITYKIT

Probably the most "scary" part of making AR is that for you to turn it into an App via Code. Some people would use Unity or Unreal, but you can actually bring your "reality" scene(s) from Reality Composer as is without issue.

RealityKit makes it so much easier!

Here is the default Reality Project .rcproject that comes with AR Template. It simply brings in the "reality" composition that you created and adding it into ARKit. You can still always edit the .rcproject.

If you wanted to bring your own .rcproject, it cannot be any easier: simply drag and drop your project and load the project by "scene" or load by object name, if you actually give name for each object asset inside Reality Composer.

Of course it would be better if you have some knowledge about XCode, adding extra additonal 2D UI and interactivity on top of your AR experiences.

Although you can actually already send the "reality" AR projects as it is without using App. Your target users can actually enjoy a bit of AR experience via AR Quick Look opening "reality" files, it has its limitations but still pretty amazing!

So, this is only the beginning of AR creations. Now it's actually so much simpler thanks to Reality Composer and 3D visual compositing editor.

There are a lot of areas to explore and discover. I am hoping that you can give it a go yourself and see if you can start making some creative 3D Augmented Reality contents and experience. You can now become a storyteller or director of AR experience simply by using iPad or iPhone.

Have fun!

A Process of Augmenting AR Human Figure

2018-01-03T01:12:00.004-08:00

It comes a long way for Augmented Reality (AR) to be adopted by everyday people. But seems like slowly we are getting there. The FUN factor needs to be there. Maybe INTERACTION and INFORMATIVE will be the next. However the key factor here is REALTIME and FUN factor.

Laying down extra informations on top of everyday visions is slowly but surely become more apparent.

AR Face Masking

We have seen a lot of apps starting to roll out realtime Face Masking, such as Snow app, Snapchat, and recent Facebook Messenger apps that allow user to use their smartphone to mask their face with all kind of interesting face masks. Some are animated, some deform or give make up, some gives some fun masks.

In 2017, Apple reveals the iPhone X. The unusual iPhone with true depth front camera. It is capable of capturing face animation and augmenting 3D render at 60 fps. Apple demonstrated the feature via their iMessage Animoji app.

iPhone X is a truly fascinating device. It is something that many will and eventually adapt. A powerful face scanner in their pocket.

iPhone X --> Blender --> 3D Talking Head

Below is Blender Sushi Guy's little experiment that uses iPhone X front camera to transfer face animation data into Blender in realtime and use it to animate CG Puppet.

Obviously this is also possible to render in realtime on the iPhone, giving the user some kind of quantum leap to become an AVATAR. Perhaps we will soon have some kind of custom avatar app and also a new social media utilizing the iPhone X front camera? We will see.

This technology is still early but has a huge potential. I am looking forward the day iPad and perhaps MacBookPro and MacPro also has the true depth front camera and perhaps also some kind of laser back camera for AR experience.

ARKit / First Test

2017-06-08T04:39:00.000-07:00

On WWDC 2017, Apple introduces its ARKit for developers, special API to present Augmented Reality (AR) contents, implemented to every iOS mobile device with A9 chips (iPhone 6S and up, iPad Pro).

I watched their first introduction to ARKit, titled: "Introducing ARKit: Augmented Reality for iOS" just today. I actually knew only very little programming and Swift code, but manage to follow their basic introduction, providing all the template and scripts like shown in the video.

Basically, for simple AR overlay, we just need to specify our own 3D model path in Swift code. Apple provides the basic Airship 3D model.

For my own 3D contents, my first test is to use contents generated via Blender 3D open source. Easy enough, I guess. A quick searching, I found out that we need to provide a specific kind of 3D format: Apple iOS expects SCN format. And apparently we can easily convert from DAE to SCN. I am not sure about other format like FBX.

It works!

Few important things to note:

1. DAE to SCN

You can apparently convert DAE format into Apple SCN to use in XCode 9:

https://forums.developer.apple.com/thread/16107
https://blender.stackexchange.com/questions/14584/properly-export-collada-for-use-in-scenekit

2. Orientation of 3D asset

Blender world axis is unfortunately always weirdly Z up, so very likely we need to rotate the object in X axis for -90 degree in order for it to correctly being displayed as augmented reality content later on inside the XCode for iOS app.

PS: Be careful with rotation of asset because it will affect all kind of interaction you give to the model later on.

3. Unit size
You want to make sure the 3D object is modeled in real world unit size (meter). I set it up in Blender so that my object is around 20 cm.

4. Good idea to understand Apple iOS SceneKit better and everything that goes around it.

http://saltpigmedia.com/blog/First-steps-in-SceneKit

http://teks.co.in/site/blog/scenekit-on-ios-8-know-the-basics/

https://code.tutsplus.com/tutorials/an-introduction-to-scenekit-user-interaction-animations-physics--cms-23877

http://ronnqvi.st/3d-with-scenekit/

https://blog.xamarin.com/3d-in-ios-8-with-scene-kit/

5. If we are using DAE, we should know the limitation.

You will soon want to incorporate animation, so that is something you need to research also. And how the animation will be triggered: continuous looping, etc, you need to also think about all that. Basic interaction trigger can be done from within Apple ARKit itself.

6. Are we going to work with 3D SceneKit, 2D SpriteKit, or fully Metal content?

This is quite intriguing and a little challenging. Eventhough we can provide some 3D assets, this will not be enough. You will soon want to think of many other concepts. It is almost that we might be needing to have some kind of authoring tools or framework on top that can setup all of these stuffs for us. Eventhough the ARKit can give you the easy want to "place 3D contents" and "track the environment", that is only small part of the whole thing: To make actual app that you are proud of and can be shared with other people!

However, this does not need to be a daunting task. I think if you manage to successfully view 3D objects in real world via your iOS device, that is a good first step. Try it in full glory, it is fun and amazing to see AR happening in front of your eyes in 60 fps from the screen of your iPhone!

7. Marker and AR
So far, I did not see any "marker" tracking being mentioned with ARKit, maybe it is possible, maybe not. I need to investigate...

NOTES AFTER FIRST ARKIT TEST

For super basic app that only displays 3D model like it is being placed and augmented in real world environment, it is pretty straight forward, almost as easy as uploading 3D model to Sketchfab. Despite needing to use the basic AR template code.

However, soon enough I realized there are a LOT more other functionalities or AR authoring functions that arise:

- Interactions you give user with the AR 3D model

- The actual interface

- Advanced lighting system

- Animation and interaction

- Particles and dynamics and physics

To make an actual app, we need to take considerations of many things:

- What are we really creating?

- Storytelling?

- Design purpose?

- Character based?

- Games and Interactions?

- Loading and preloading of 3D assets?

- Are we going to utilize other authoring tool like Unity or Vuforia?

For now, I will probably give a few basic tests of 3D models, maybe try making a setup that uses better lighting, testing the quaility of model, etc.

PERSONAL PORTFOLIO APP

Any 3D artists wanting to make their own little app to showcase a simple 3D portfolio, this can be a good fun way to do it. Especially if you are making 3D assets, animations for games, in real time. Can be a little fun side projects. You need to be paid Apple Developer if you want to put your app inside App Store, but if I am not wrong: free Apple Developer account do actually enables you to use XCode for free and kind of deploy on mobile anyway.

Ideally, a really good service like Sketchfab can provide App that takes advantage of Augmented Reality. I guess, before long, we will be seeing more AR apps, and users will start to get comfortable of seeing virtual objects. And soon enough we will have Apple AR Glass of some sort. Probably Apple Iris, as it was rumoured being called on Reddit thread last week. Who knows?

Think of it this way: you have TODAY the ability to place AR contents easily and present it to anyone with iPhone and iPad Pro. This will open up a lot of possibilities and ideas. Surely, AR is not new and it is growing, but it is a very exciting now that Apple opens up its mobile ecosystem to this.

AR / 3D Selfie

2014-07-15T23:48:00.001-07:00

This is the trigger image, download Aurasma app and point directly to the image.

I am back! Sorry for not posting for a very long time, I have been busy on other things. I am starting to have big interest in creating Interactive 3D, Game Programming side of things.

I was curious to test how Aurasma is going and I think it's about time I came back to think about Augmented Reality (AR) and the contents. The technology has improves just slightly.

This time one got some new set of tools and skills:

Unreal Engine 4
Unity 3D
iPhone 5S and iPad Air
Structure Sensor (for 3D scanning)
Kinect 2 (for 3D scanning and motion capture)

AR 3D SELFIE?

Above AURA will trigger a my "3D Selfie". (I promise I will NOT do this often!)

You probably don't like the idea of selfies photography too much :) However, I am predicting that it will be the trend in the next few years to come. Before 3D holographic selfie (with sound) came out.

Actually, I quite like the idea of keep 3D memories portrait of people digitally. A bit Star Wars-ish.

TOOLS NEEDED:

1. Blender 3D
2. Aurasma App
3. 3D scanner (software or hardware based)
4. Computer + Internet

LATEST AURASMA AND THE AR PROCESS

Aurasma still uses DAE Collada as the AR container for 3D content. By today, I was hoping they would improve to FBX and allow for procedural material, etc. Anyways, I think that's pretty sufficient, still, for most simple 3D contents. A limit 15000 tris is kind of alright I guess.

1. Capturing 3D data
For 3D scanning, I am using the Structure Sensor on the iPad and "itSeez3D" app.

Although Structure Sensor provides other app such as Structure app, Skanect that can scan a slightly higher resolution, at this point, only "itSeez3D" can really capture the 3D data as well as the texture.

Having a portable 3D Scanner like this is really handy, no longer we need to be stuck next to 3D scanner. We can simply go outside and scan any objects. I supposed 2014 is really the year when 3D scanning and 3D depth capture is maturing.

ALTERNATIVE TOOL FOR 3D SCANNING:

Hardware based 3D scanner.
Alternatively, there is a more expensive 3D scanner for more serious job:
http://www.artec3d.com/hardware/artec-eva/

But I think Kinect 2 is going to be great for 3D scanning and motion capture with higher resolution.

For software based:
You can use 123D Catch from Autodesk on the iPad or iPhone, that still works alright. I will not rely too much on Autodesk though.

At some point, Microsoft also has some research on 3D scanning in term of point cloud, now under name Photosync. This is for bigger scale of object.

There is Photosculpt that can turn 2 stereo images into 2.5D 3D mesh. This gives you nice displacement and normal map:
http://photosculpt.net/buy-now/

Depending on the need, I think the middle ground is to have something like Structure Sensor. Not too high resolution, not too expensive, can be used to scan 3D data that's good enough for AR creation and 3D printing. I guess the tool will get better. For now, I think this is the most ideal.

2. Processing and preparing 3D content using Blender.
I am using Blender 3D to put everything together. There is no need for commercial programs like Maya or 3dsmax. Blender is the perfect tool.

In addition to Blender, MeshLab is also good to clean up scan data.

It is basically a simple process:
- Import the OBJ model scan
- Assign the texture, ensure we are NOT in Blender Cycles environment, the Material needs to be setup inside Blender Internal Render (the default renderer). Doing this will ensure the Object + Material + Texture + Light will work correctly when exported as DAE.

The cool thing in Blender is that one can easily reduce the number of triangles and triangulate the model using the Modifier. I reduced the number from 50000 tris -> around 13000 tris. I remember some times ago, when Blender does this, it does not keep the Texture and we need to do some baking. However, we can do this so easily now. Thanks Blender devs!

Using Blender to prepare 3D AR content

3. The rest of the setup is all mentioned inside Aurasma documentation.
If this is your first time to create 3D Contents, I recommend you to read the Aurasma Documentations.

Pack the needed content and compress it as TAR using 7zip.
- 3D DAE
- Thumbnail
- Texture
- MP3 (if audio is needed)

4. Upload and set up the AURA using Aurasma Studio online tool.
Once everything uploaded. We can check the result using iPhone or iPad. All done!

Ready to be shared to the world.

NOTE:
Now, there seems to be an issue of iPad Air detecting the Aura. I am not sure why, but it works instantly and really fast on iPhone 5S. But my iPad Air is detecting nothing! Very strange. Anyhow, I found that the AR reading performance is a lot better on iPhone 5S.

ADDITIONAL NOTE

Last note, so far I am testing a straight on Augmented Reality just because most of my reader will be seeing the content from computer screen. AR is of course, 3 dimensional. So I will try to add some turntable animation, if the trigger image is presented flat on the wall or computer monitor. And as well providing another trigger image that is flat on the ground, where the 3D AR is projected vertically.

Something to keep in mind.

There are plenty of areas that still unexplored. Opportunity for 3D artist to came up with lots of interesting 3D contents. I will try to experiment with this more. Aurasma supported 3D Bones and looping animation, so I think that's a pretty big deal. I might create some abstract 3D procedural animation works using Blender. We will see.

I really recommend you to give this a try!

AR / Racing Cars

2012-10-07T15:35:00.000-07:00

This is certainly an impressive demo of AR capability when it gets more realistic. I imagine in near future, the AR could use the environment lighting from the actual environment to get even more realistic result.

The McLaren P1 AR is using real-time environment reflection and fake shadowing under the car, and this seems to double the realism:

https://www.facebook.com/mclarenautomotive

McLaren P1 AR Demo App for iOS devices.

The paper, the blurry environment, and overlapping with real creature can kill the illusion of realism.

This is the one I made and published using Aurasma, minus realtime reflection, minus any fancy mapping (just Diffuse Texture Mapping), and no fake shadow.

The Alesi model was provided by Bruce Allan, this is an experiment test AR to see the capability of Aurasma.
http://www.triggar.com.au/

It could certainly be better with additional mapping and fake shadow. I don't think Aurasma is supporting real time environment reflection that follows the gyro yet.

I supposed, faking the environment lighting and shadow with occlusion will be the way to go for now.

AURASMA x 123DCatch / Truck at The Grounds

2012-09-14T16:39:00.002-07:00

Requirement:

Aurasma Lite App for iPhone/iPad or Android

Click on this link to subscribe to my Aurasma AR Channel: (one time only)

http://goo.gl/wRlbw

(future AR contents will be automatically loaded once you subscribe to my Aurasma Channel)

AURA:

Truck at The Grounds

123DCatch Process

This is another test I did, the process is pretty "brute force", where I actually take photos using the iPhone while walking around a truck. I took around 80-100 degree view of the truck and convert that into 3D.

Clean up and poly reduction of the 3D truck mesh are all done inside Houdini FX. This is to ensure that the 3D mesh is good for Aurasma. Quick lighting and export to DAE via Maya.

Something to keep in mind: Keep 3D object small under 5-10 MB. Depending on TAR compression of Augmented object and also complexity of the DAE, the loading of the Augmented content may take a while the 1st time around.

Aurasma does a great job displaying 3D contents, as usual, if during authoring everything is smooth, then your AR content will be displayed without any hickups.

LIST TO DO

Rotating 3D
Interactivity
Animated 3D Face
3D with Sound
Full 360 3D object
Particles and FX

AURASMA x 123DCatch / Lion Statue

2012-09-08T05:27:00.005-07:00

Requirement:

Aurasma Lite App for iPhone/iPad or Android

Click on this link to subscribe to my Aurasma AR Channel: (one time only)

http://goo.gl/wRlbw

(future AR contents will be automatically loaded once you subscribe to my Aurasma Channel)

AURA:

123DCatch Lion Statue Outside Chinese Garden, China Town, Sydney, Australia

NOTES

For this Aurasma 3D Overlay, I used a 3D mesh "scan" which is created using iPhone and this 123DCatch app from Autodesk Lab. I took few shots using iPhone on Saturday morning and the result is pretty amazing.

The original mesh OBJ is rather large, around 20000 tris. I usually use Blender for polygon reduction because the result is fantastic. But Blender poly reduction will usually gets rid of the original UV.

HOUDINI COMES TO THE RESCUE

This time I am using Houdini to do poly reduction of the mesh and I think it is quite cool that Houdini allows to keep the UV information from original while reducing the mesh with realtime result.

This is 50% polygon of the original mesh.

For AR, efficiency is everything. A really good 3D object for AR is as efficient as low polygon modeling for game engine in the past.

ORIENTATION

I am doing export import using Maya, Blender, and Houdini. Usually Maya is the one doing the final export for Aurasma. Maya's coordinate is Z+ axis as Front Facing.

Inside Aurasma, we must decide how the 3D AR Overlay is going to face:

Front-On the Trigger Image
Standing on Trigger Image (allows more viewing area, surrounding 3D object)

So far, for this blog, I am overlaying 3D that is Front-On to the Trigger Image. So, the actual 3D object should be facing the Y+ axis.

AURASMA / Kraken Aura

2012-08-29T15:54:00.002-07:00

Requirement:

Aurasma Lite App for iPhone/iPad or Android

Click on this link to subscribe to my Aurasma AR Channel: (one time only)

http://goo.gl/wRlbw

(future AR contents will be automatically loaded once you subscribe to my Aurasma Channel)

AURA:

PROCEDURAL KRAKEN DANCE

This is the trigger image.

3D Model was created by Umberto D'Andrea.

Rig, animation and 3D animation for Aurasma is created by Jimmy Gunawan (me).

Thanks also to Suchan Raj for rigging direction.

Resulting Aura:

AURASMA / Digital Finger Puppet AR

2012-08-19T23:35:00.001-07:00

Requirement:

Aurasma Lite App for iPhone/iPad or Android

Click on this link to subscribe to my Aurasma AR Channel: (one time only)

http://goo.gl/wRlbw

(future AR contents will be automatically loaded once you subscribe to my Aurasma Channel)

AURA:

TEAPOT GUY

Print out Image Trigger:
https://dl.dropbox.com/u/3654459/PUPPETAR/teapot_guy.pdf

This is my first test on Aurasma Sequence Overlay advanced feature. This gives user the ability to create non-linear sequence of the Augmented Reality contents or the overlays. So, you have multiple overlays that you can control by tapping the screen of the iPhone.

You could actually have a slightly sophisticated interface that allow triggering of action or animation. I found that even for this the authoring process is really simple and easy.

With character above already created and rigged, I could easily create more animation, apply different trigger image, and new AR can be published and ready for users to see within minutes.

For this AR, I am actually trying to make some kind of Digital Finger AR Puppet. I remember creating finger puppets like these back in elementary school, teacher does not really like it :) Anyhow, now the time is different, I think Digital Finger AR Puppet is actually a good idea.

I will do more of this in near future. Probably with additional sound, interactivity.

NOTES:

1. Trigger image in form of "photograph" with a lot of details and good contrast on certain area works better. Above image is not the best image for triggering because it has a lot of flat colors.

2. Image detection and tracking are a lot better on the iPhone, compared to the iPad. Even on iPad 3. I believe Aurasma will improve it overtime. Maybe even better on iPhone 5!

3. iPhone and iPad focusing distance matters. You definitely need to make sure that your trigger image can focus. A1-A2 Poster is probably the most ideal. Usually the way I know it will work properly is if the user can cover the iPhone screen with the trigger image, 90% it will trigger the AR.

4. Lighting condition needs to be controlled and ideal for AR image to be detected and to trigger, such as indoor lighting with ambience, no highlight or strong shadow or not in the bright space.

The best condition for AR Aurasma trigger image is actually LCD computer screen with no reflection. So, if you are reading this post and you want to test the AR, just point your iPhone or iPad to the laptop monitor or your computer LCD Monitor.

VIDEO DEMO:

ADDITIONAL NOTE (Update 2012.09.10):

I really believe that Digital AR Finger Puppet or AR Hand Puppet can potentially be a new trend for years to come. I mean, with mobile phone always stays next to your hand, what else is best to actually "trigger" AR than your very own hand?

In the past we have seen Winsor McCay performing and talking to "Gertie the Dinosaur", which was basically a hand drawn projected animation. I think it is about time somebody does the same with AR character.

In a Japanese manga, we have something like this:
http://en.wikipedia.org/wiki/Midori_Days

Perhaps one day AR character can have personality, such as "Sumomo" character in "Chobits" anime.

There are plenty developments in Japan already on games/software/apps that utilizes AR for CG character such as "Love Plus' app, and also in recent Hatsune Miku (popular Japanese idol) AR.

Eventually we could design our own unique, complex, personalized AR character. In the meantime, app and tool like Aurasma Studio by Aurasma is a perfect starter tool to create your own 3D AR virtual character!

AURASMA / Dr.D Mouse Pad

2012-08-13T05:09:00.001-07:00

What other object do you think can be "augmented"?

AURASMA / Mario AR

2012-08-09T04:50:00.001-07:00

Requirement:

Aurasma Lite App for iPhone/iPad or Android

Click on this link to subscribe to my Aurasma AR Channel: (one time only)

http://goo.gl/wRlbw

(future AR contents will be automatically loaded once you subscribe to my Aurasma Channel)

Example AURA:

SUPER MARIO PUSHUP

Once you are connected to my AR channel, and have Aurasma Lite app, simply point into the image above.

For a flat rendered animation which was original 3D model, this actually creates quite a nice presentation, even though it is really just flat piece, because you are getting the full 3D presentation of the objects, instead of low resolution 3D.

AURASMA / First Time Aura

2012-08-01T18:37:00.001-07:00

I tried Aurasma Lite app on iPhone and iPad long time ago that allows user to quickly create AR contents (video overlay, image overlay) based on a photo taken and uploaded from iPhone or iPad. This was one of a kind and I did a few test on it. It actually works quite well. You can quickly take video or photo (as AR overlay) and attach another Photo to trigger that contents which can be based on Location as well.
http://www.aurasma.com/what-is-it

The problem I found in the past to create "Aura" contents was the Internet bandwidth for uploading and also authoring on iPhone/iPad is not the most convenient. I would rather do the authoring on Desktop computer at home or studio.

AURASMA STUDIO TOOL
Recently, Aurasma released their online authoring tool for Aurasma Channel and Content Management which I think a brilliant move and so I have more interest in it again.

Finally I have chance to test it today.

I have previously tested:

Junaio AR Creation Channel >> limited usage and upload, it has similar AR Content Management to Aurasma, but I think Aurasma take it to higher level.
AR Media >> contents downloaded as *.armedia, can be viewed as still 3D on iPad and iPhone

Aurasma on TED

QUICK TEST TRIGGER IMAGE
Like many AR authoring tool, they would usually be quite picky with the trigger image. Apparently the more details you have on the photo (texture, color), the better it can be tracked and to trigger your overlay AR.

Below is my first test:

This TRIGGER IMAGE will not work.

Grey-ish black and white simple image is not a good trigger image apparently.

So I tried similar image, but this time with crazy color like below:

This TRIGGER IMAGE actually works.

The colorful image works! If you want to try what image behind above image, you need to have Aurasma Lite app, and subscribed to my channel.

GOOD TRIGGER IMAGE FOR AURASMA: (FROM THEIR DOCS)
A good trigger image can be described as one with plenty of visual variation, making it distinctive.
- Images with lots of different shapes are distinctive, but images with lots of plain or empty patches are not:
- Images with lots of texture are much better to use than those with simple blocks of colour:
- Exact, self-repeating textures often result in poor tracking, as lots of parts of the image look like each other and are hard to tell apart:
- Continuous or repetitive text is not ideal for tracking.
- N.B - Trigger images exhibiting either of the last two characteristics in this list can sometimes be improved with the use of masking, provided that there are other areas of the image which are distinctive.

If you are outside, some buildings can make good triggers, although extra care should be taken.

How to take a GOOD shot:
1. Stand a reasonable distance away, but so that the building still fills the image. Try to be as square-on to the building as possible.
2. Flat buildings are good.
3. Look for distinctive features (the corners of the windows).

How to take a BAD shot:
1. High-rise buildings don't work very well, unless you just want the first few floors. Remeber to be square on.
2. Buildings with too much depth are bad. 3d scenes like statues, open areas, trees, arches are bad - really you should be aiming for buildings with a flat facade
3. Make sure the building fills as much of the window as possible, but that you still have all of the building in shot. Too tall and thin is bad. E.g., Big Ben is too far away to be recognised well .
4. Occlusion of the building is bad. and look out for excessive shadows.
5. Try to choose scenes where the background (and foreground) is not likely to change too much. Ask yourself: “Is the user going to be able to take the same photo as me?”
6. Avoid buildings with too much glass or reflective surfaces.
7. Avoid too many identical features (e.g. bricks/windows).

AR CREATOR/AUTHOR SIDE

Aurasma online authoring tool is called "Aurasma Studio":

You get access to Aurasma Studio only after you register to Aurasma and confirmed as Partners. This is probably to prevent random users uploading random unwanted contents to public.

http://www.aurasma.com/partners/

I got verified as Partners just few days after I registered.

Inside the Aurasma Studio, you will be presented with beginner step by step tutorial (HELP section):

First time user should definitely check this out. Once you successfully upload your first Aurasma AR content that works, the future content creation is really easy.

Aurasma makes it really simple for users to setup your AR contents, you are presented with these terminologies:

Channels
Trigger Images
Overlays
Auras

If I put them all together with my own word, the step is like this:

You create your own Channel. Just like YouTube Channel or TV Channel, except that this is YOUR own AR Channel. You are allowed to create multiple channels.
You prepare Trigger Images and Overlays. Trigger Images is basically a still image that will trigger the AR contents. Overlays is the AR contents, can be in form of another image overlay, or can also be video overlay and 3D overlay.
Auras is a complete package of Trigger Images and Overlays that you can put on your Channel.

The process is as simple as it gets. Compared to any other AR authoring tools available out there, this one is definitely simplest one.

You can set your AR Channel or Auras contents as Private or Public.

ADVANCE: 3D Aura / 3D AR Content Overlay

I copy paste this from Aurasma 3D content overlay documentation if you are curious on the 3D model and animation requirement:

****************************************************************

3D Models for Overlays

Quick Overview

To upload a 3D model, Aurasma Studio requires a file to be in a .tar format, which can be created with a number of computer programs. We recommend 7-Zip for Windows and GUI Tar for Mac.

A .tar file should contain:

The exported/scene model using OpenCollada (.dae file)
All textures referenced in the scene file (.png format)
.png thumbnail (256×256 pixels), named thumbnail.png
.mp3, if sound is required

What formats do you support?

Our 3D model support is based on the Collada format (v1.4 or v1.5), as exported by Maya 2012 and 3DS Max 2012 (OpenCollada Exporter).

Uploading to the Aurasma Studio

Open the Overlays screen, click Add, and select your .tar file (don't click on Bulk Add).

Due to the processing limitations of today's devices the first generation of 3D models should meet the following criteria:

Models

Under 10,000 triangles.
Scene should be centred around the world origin (0, 0, 0) to ease placement.
Meshes must not have any non-uniform scaling.
Exported meshes must be triangulated. (This is usually handled automatically at export stage if the option is selected.)
No more than 20 objects (meshes) in a scene. Animated rigid geometry is less processor-intensive than skinned geometry, but extra draw cells are more intensive. (Ultimately, once you have passed 10 or so objects, it is something that needs to be looked at on an individual basis.)

NOTE: Consider what you're making and how much screen space it will take up - you won't need 10,000 triangles and a 1024 square texture for an object that may only be 3cm tall on an iPad screen.

Textures/Shaders

One material per mesh. If you need two materials then use multiple meshes.
Individual texture maps (.png format) must be of a dimension to the power of 2 (we recommend 128×128, 256×256, 512×512 or 1024×1024). Textures cannot be larger than 1024×1024 pixels.
Limit your maps to no more than 2 per scene. For optimised loading times, reduce the number of maps. The use of smaller, tiled textures is fully supported and recommended where appropriate.
Emissive, normal and diffuse maps are supported, as well as specular colour (but only when using the OpenCollada exporter).
Use the simplest shader possible for each situation. The order of complexity is as follows: constant < lambert < phong < blinn.
Alpha blending is very processor intensive. If you use alpha on sections of mesh, put it in its own shader. Alpha on solid objects is not recommended and is better suited to planes.
Material names must not contain spaces, brackets etc.
Material names must not start with a number (OpenCollada characteristic).

Rig

Bone names must not contain spaces, brackets or other special characters — replace spaces with underscores if needed.
The export rig should contain a maximum of 100 bones to optimise playback — the performance hit comes largely from skinning.
It is advisable to name all bones properly.
Objects cannot be linked/parented to non-export objects — constrains are fine.
Do not scale anything.

Animation

All animated objects/controllers must have a key on the first and last frames.
When exporting a .dae, you must bake and resample keys in the Collada options (see above).
No animated materials/material parameters.
Scaling is correctly supported in animation.
No animated light colour.

Lights

Scenes must be lit. 3 lights are recommended. Breaching this will cause performance slowdown on less powerful devices.
Omni/point/directional lights are supported — we recommend point/omni lights.
Light value is based on the light's colour value, not intensity, e.g. mid grey = 0.5 multipler.
Ambient light is supported, but may cause overexposure to scene.

With the constant technological development of smartphones, yet more complex 3D models will be supported by Aurasma.

****************************************************************

FIRST TIME 3D AR
Ok, so I followed the requirements above to upload 3D content, and to my surprise the process is really smooth.

Open Collada Export (DAE file format) is quite universal and accessible on many 3D programs, and so I manage to attach my 3D Sadako model with animation (from previous post, modeled and textured in Blender, animated in Maya) into Trigger Image.

I use one of the Sadako 3D posters to trigger the AR content, but using Aurasma Studio I could easily attach another new alternative Trigger Image.

So, if you want to try this yourself, simply Google Image Search: sadako 3D poster. One of the poster will trigger Sadako AR to appear on your iPhone or iPad Aurasama Lite App. Remember: you need to subscribe to my channel first. Link below.

Alternative Trigger Image for Sadako:

USER TARGET SIDE

Once AR contents or Auras created, you then SHARE your Channel address using link.

My Channel link looks like this:

http://goo.gl/1CFPS

So, supposedly the target users click on that, it will make a jump to download and install Aurasma Lite app on their phone and then adds this Channel for them to subscribe.

Once all set and good, users can then view the future Auras from the subscribed channel. All they need to do is to point their mobile phone into Trigger Images and AR contents overlay will show up.

In summary, minimum requirement for target users:

1. iPhone, iPad

2. Aurasma Lite App

3. Internet connection

ANOTHER TEST

If you point the Aurasma Lite app, you will see a video animation (flat) overlay on top of this image.

CONCLUSION

This is a very simple test, my first time experience using Aurasma Studio.

I actually quite like it and found this to be highly potential for a lot of things that are not yet explored widely by public. I am going to try all kind of AR contents using Aurasma. Maybe for now, Aurasma is really the perfect AR tool for me (and you) to publish all kind of AR ideas I have in mind. Let's give this a try!

AR / Sadako The Ring

2012-07-12T19:41:00.000-07:00

iOS Version: (use ARMedia Player app)

https://www.dropbox.com/s/mzp8pmxnyd1s4r0/sadako_ios.armedia

Webcam Version: (use ARMedia Player program on your computer)

https://www.dropbox.com/s/ht8jajzc7belcfx/sadako_webcam.armedia

AR Marker for Sadako is here:
http://www.inglobetechnologies.com/en/products/arplugin_su/download/Marker.pdf

How to use:
Simply point your webcam or iPhone/iPad to the marker.

NOTES:
ARMedia has random limitations such as:

iOS version is a still 3D model only, no animation. Can be rotated, scaled, but that's all.
Webcam version can include the animation, a single light, video and sound.
B&W Marker only, would be nice if ARMedia allows colored image tracking.
I still just use the ARMedia default marker, just because their AR Marker creator is not working as smooth as expected.

UPDATE 2012.11.15:

Below is Sadako Trigger Image if you are using Aurasma: (make sure you subscribe to my Aurasma channel first!):

EXPERIMENT / AR Ink Marker

2012-07-12T15:27:00.002-07:00

This is an old AR work I did in the past, when Autodesk 123DCatch was still called the Project Photofly.

I did a scan (using Photofly aka 123DCatch) of this "pig bunny" doll of my niece, brought it into Blender and use Junaio Creator to augment the 3D still model on the palm of my hand.

This was a really quick to do, probably can be a concept used in the future.

Welcome to PuppetAR!

2012-07-12T05:17:00.004-07:00

Thanks for visiting the PuppetAR blog!

This is yet another blog I opened to share some of ideas, experiment, and all kinds of things that pop from my head. Here at PuppetAR, I am going to write and share everything related to Augmented Reality.

BACKGROUND HISTORY
At heart, I am a CG artist generalist, based in Sydney, Australia. My 3D tools are Maya, Houdini, and Blender (I wish I had time to learn all 3D software and tools! Looking forward to learn Unity as well, it is in my task to do).
My main area of focus are as character rigger, animator, and effects artist.

WHY INTEREST IN AR?
Apart of doing 3D and CG works, previously I have studied graphic designer and worked as graphic and multimedia designer, doing print and web layout and designs. So, interactive design is always part of my passion in this area, AR and 3D is the next stage of my exploration.

I have been following AR for years since it was first appear in the news. With the advance and development of Augmented Reality (AR) over the years (with Google Project Glasses also become a reality), we have come to a stage where human and computer virtual interaction has become a natural everyday thing.

AR is really a perfect media for people to deliver a new kind of interaction that has been dreamed for years. To have special additional layer of reality (not just text information HUD), where one can virtually interact with 3D object and creatures is really one of futuristic dreams came true.

Unlike CG for film, AR is more immediate. It is rendered in real time and users can interact with it, such as what people experiencing inside games.

This all is really interesting for me because the possibility is endless.

WHAT PUPPET AR BLOG IS ALL ABOUT?
I don't know the answer for this yet, to be honest :)

Maybe I will design all kinds of virtual 3D AR puppets or characters that people can download and play with on their mobile phone. Or maybe I will do some kind of story telling or experimental AR works? Maybe make animated AR as popular as animated GIF today (looping 3D)? Probably it will contain some rough ideas or projects that I have yet to do? Can 3D augmentation delivers a new user experience that can be emotional?

Perhaps I have interest to do some kind of Virtual Mechanical Toy that anyone can control, I have always big interest in robotic and things like Automaton, as well as Game Development.
http://en.wikipedia.org/wiki/Automaton

At some point in the past, I draw a lot, I make comic, and I design original characters. I also feel like doing some interactive comics or children book. It kind of "stuck" when I get to learn the technical side of computer graphics, 3D, etc. Now that I gained a little bit more skill, I feel like doing more creative works.

There are so many character design I wanted to make, and maybe little by little I can share them in form of Augmented Reality. Let me know if you have AR ideas you like to create, maybe I could help in the artistic or technical side.

Anyways, the main focus of this blog articles will be exploration of Augmented Reality technology, mixed with:

Character Design
Character Rigging
All kind of available tools (for 3D, I use Maya, Blender, Houdini)

Ok, maybe I get over passionate about it, but that's always a good start :)

EXAMPLE CASE: GLORAR
Recently there are more AR apps appearing and many news related to AR.

One app for iPad called GlorAR, for example, allow user to easily upload and placed AR in a world map. I actually tried it all around Sydney, Australia (where I live):
http://glorar.com/

Since the GlorAR app was only on the iPad, I have to walk around with iPad. It's very awkward but there is an interesting sense of immersiveness while seeing 3D augmentation in 3D (I nearly got hit by a car or bump into people).

EXAMPLE CASE: AURASMA
Aurasma is one of most interesting AR app available out there already. You could easily apply AR object (video, 3D) into any image.
http://www.ted.com/talks/matt_mills_image_recognition_that_triggers_augmented_reality.html

MARIO 3D AUGMENTED REALITY
As a start, here is a Mario 3D model (simple model based on original Mario, no animation, just still pose) created in Maya that you can download and view using AR Media app on the iPhone:
https://www.dropbox.com/s/huyi0anppassv0q/mario_pose_001.armedia

How to view this using iPhone or iPad:
1. Download and Install AR Media app from the App Store (for iPhone, iPad).
2. Download and Print this marker below (or just on laptop screen)
http://www.inglobetechnologies.com/en/products/arplugin_su/download/Marker.pdf
3. Run the AR Media app and view the AR Marker to see the 3D Mario.

Augmented Mario from a marker on iPad on the stair of my apartment.

WALL OF INSPIRATIONS

http://www.boring3d.com/
http://www.kidqube.com/
http://www.robives.com

Marco Tempest - The Original Digital Illusionist
http://marcotempest.com/