Intro To Unity3D and AR (Augmented Reality)
Learn Unity 3D & C# – From Basics to Advanced
Unity3D is a powerful suite of tools (Project IDE, Code IDE, run-time) for game development. As always, RivelloMultimediaConsulting.com/unity/ will be the central location for deep articles and tutorials, Facebook.com/RivelloMultimediaConsulting (like us!) will engage the growing RMC+Unity community, and for the latest opinions and cool links follow me at Twitter.com/srivello. There is incredible momentum in the Unity3D product and its community. Here is more information on using Unity3D with Augmented Reality.
Augmented Reality
Augmented reality (AR) is a field of computer science that involves combining the physical world and an interactive, three-dimensional virtual world. I originally wrote a complete demo, source code, and article for my client Adobe – “Augmented Reality With Flash“. Now I’ve updated the information regarding Unity3D. In a future article I may make the full Unity demo source code available. While mainstream audiences are now becoming aware of AR, it is not new. Its background is intertwined with decades of computer science development. Virtual reality (VR), AR’s more familiar counterpart, is the replacement of the user’s physical reality (particularly that which is experienced through sight and hearing) with a computer-generated reality. The idea of a virtual experience is exciting — creating entertaining and educational sensory encounters that do not exist in our everyday lives.
From a consumer standpoint, it seems that AR advances have come out of nowhere to surpass VR advances. The acceleration in AR technology is due to two major factors: First, users are still experiencing reality, so believability is easier to achieve. Adding simple graphics (such as text or simple shapes) and color effects (such as night vision or thermal vision) to reality creates a better user experience. The user is still seeing a mostly familiar world. Second, this more subtle use of computer graphics is less expensive with today’s technology, making it more feasible than VR. Let’s take a look at augmented reality, its current uses, and its future potential.
Practical Uses of AR
The video game industry has released major augmented reality products for more than a decade. The Eye Toy for Sony PlayStation 2 and 3 takes input from an inexpensive video camera and composites the live video feed with CG onto the TV screen. This toy detects the user’s body position in front of the camera as an alternative input method to the typical joystick or gamepad, deepening the user’s immersion into the game world. Detection schemes continue to advance, allowing for more engaging interaction.
There are AR applications outside of console games, including military and consumer products, too. Night-vision goggles and targeting-assistance technology help marksmen in battle, and children’s stories come to life with AR-enhanced books. The uses are vast. With AR-enhanced books, each page of a real-world book is outfitted with a marker — a pattern that is detectable by AR. As the child turns each page, a computer is able to place a virtual 3D animation on top of a physical image printed on paper. While the marker is often an image as in this example, it could also be the user’s face.
Basics of AR
- Marker Detection – The marker is located by the application. Typically the device webcam is used. The webcam inspects the physical reality near the user to find a predetermined marker (image or the users face).
- Transform Mapping – The transform (position, rotation, scale) of the marker are interpreted. Move the marker and the transform updates.
- Rendering – Now the 3D model is updated to rendered to the screen.
Essential Libraries
Because of the growing popularity of AR, many platforms have 3rd party packages (free and premium) to facilitate development. Unity too. While you could create you own system from scratch, it is highly recommended to find the best 3rd party library and use it. Your end product will be higher quality and you will save greatly on time to market. You can choose any one from this list; Unity3D Packages for AR
- Catchoom
- D’Fusion*
- Metaio
- NyARToolkit (Japanese)
- ObviousEngine
- PointCloud
- SeeingMachine’s FaceAPI
- Sphero
- SSTT
- String* – (Tutorial)
- Vuphoria* – (Tutorial)
- Xloudia
*Seem to be popular
Vuforia is the best and free to use. I tried with Metaio, it’s easy to use than Vuforia, but the tracking quality don´t convince me. Vuforia handles the pattern occlusion is robust, and it’s framerates are high, much higher than Metaio and NyARToolkit. – 3rd Party Blog Comment
There are pros and cons to each. In a future article, I may evaluate each of these options. Some of the key differences are cost, ease of use, completeness of documentation, breadth of features, and accuracy (speed) of tracking the marker from the camera and transforming the model. Each package surely has its own workflow and API, however the major concepts are the same. So once we choose a package, we follow this the basic setup.
Developing Augmented Reality
Marker Setup
Figure 1. Marker Image
The marker image is a graphic drawn, printed, and shown to the end application as it runs. Your AR package, with help from the marker data file and parameters file, will detect this shape via webcam. To design a graphic, fit the desired shape within a white square that is centered within a larger black square. Keep your shape simple for best results. Note the sun shape in the graphic. This helps the application detect which way is up for the marker and assists its angle and rotation detection. Your design doesn’t need to have this shape per se, but it should be asymmetrical in some way for best results. Webcam Setup Unity features webcam support. As the application runs and a camera is detected, Unity will show the video onscreen. This functionality may or may not be included in the AR package you choose. AR Package Setup
Figure 2. Marker Detection
A cornerstone of this project is marker detection. As the marker graphic is detected via webcam, its position, rotation, and scale in the user’s physical space are calculated. This calculation will be used in the next steps. The AR package will search each still-frame of webcam footage for the predefined marker graphic (Pattern object). The detection scheme uses just the latest still-frame of video at any given time. Model Setup
Figure 3. 3D Model Setup
The 3D model is loaded and inserted into the 3D scene, and the view into that scene. Luckily 3D is a strength in Unity and much of this work is all done for you. In the final project, this view is layered on top of the 2D webcam Video object shown earlier so the composite (See Figure 4.) of 2D and 3D will look believable. Repeated Loop: Detect-And-Update
Figure 4. Complete Process
Generally speaking your loopToDetectMarkerAndUpdate3D function will be called 60 times per second. Each frame the latest frame of the webcam footage is grabbed. The AR package runs detection on the frame. If the marker is detected the AR package will update the position, rotation, and scale data. This modifies the 3D model so it appears to be at the same position in the user’s physical space as the marker. The resulting effect is nothing short of amazing for first-time viewers.
Great Slideshow on Unity3D & AR
[slideshare id=13088203&doc=unityar-120526151111-phpapp02]
Want More Video Tutorials?
We are currently creating tons of new training content for Unity3D. [polldaddy poll=7125741]
