How to make Mixed Reality films through HTC Vive?

How to make Mixed Reality films through HTC Vive?

Date: 2018-02-02
Views: 138

Maybe you're a developer who wants to make cool commercials for your HTC Vive game. Or you are the game anchor, want to bring the high quality VR live experience for the audience; Even you are a fan of VR. Anyway, if you're interested in MixedReality, go ahead. This article describes how to use HTC Vive to shoot high quality MR video.

What is Mixed Reality(Mixed Reality) film?

Mixed Reality film is a film that is synthesized through the images in VR. Mixed Reality video is more suitable for live-streaming and propaganda films than the VR films recorded in general methods.

If it is still not clear, you can see the following film, which is the film effect produced by Mixed Reality.

http://v.qq.com/x/page/x0194621p8b.html

Make the props for the Mixed Reality film.

1.) green screen environment

2.) camera/camera (recommended parameters: 1080p/60fps)

3.) 4K screen: in order to produce 1080p video, the screen needs to be able to fit multiple 1080p Windows (recommended parameters: 4K/60Hz)

4.) HDMI distributor.

5. The HTC Vive group.

6.) the third HTC Vive handle (buy website: https://www.htcvive.com/cn/accessory/controller/

(7) VR Ready computer advice parameters: the operating system: Windows 7 SP1, Windows 8.1, or Windows, processor: Intel ® i5-4590 / AMD FX 8350 equivalent or greater, memory: 4 GB of RAM, graphics, NVIDIA GeForce ® GTX 970 / AMD Radeon ™ R9 290 equivalent or greater)

If you want to cast the picture directly to the audience, you may need to prepare the following items:

(you can refer to figure 1 for the hardware configuration of MR movies)

   

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


(1) video acquisition card: you can collect the image of the camera or camera to the computer. If you use the webcam to input the image signal directly to the computer, you do not need this device.

2.) 4K screen: in order to make a 1080p movie, the screen needs to be able to hold multiple 1080p Windows (recommended parameters: 4K/60Hz)

3.) screen (for cinematography) : to make MR films richer, it is more interesting to follow the mobile scenery of the experiencer than the fixed view. Therefore, it is necessary to play back the resultant results and assist the photographer. This device is not required if fixed viewing mode is adopted, such as network broadcast.

4.) screen (audience viewing purpose) : this screen is used to display MR images in real time, which is suitable for public display and other activities.

MR image output description (please refer to figure 2)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


In general, when VR games executive, screen window will display head perspective, start in MR images, the output of the screen window shows the four quadrant of the screen for the user to synthesis, the four quadrant images are:

. Foreground (upper left)

The foreground screen shows the visual Angle of the virtual camera. In the game scene, all objects between the head and the virtual camera are displayed in this picture.

. Foreground alpha shield (upper right)

This screen is the foreground alpha shielding, mainly used for the post-production of the film can be used as the foreground of the material.

. Background (bottom left)

The background screen shows the visual Angle of the virtual camera. In the game scene, all objects that are in the background of the head and distant background will be displayed in this picture.

. Game screen (bottom right)

How to record MR images?

In order to start the output of mixed reality screen, the following three conditions must be met:

A.) the content program must be developed by the Unity engine SteamVR plug-in version v1.0.8 (or updated version).

(B.) place the file externalcamera. CFG (plain text file, extension, CFG) under the content program execution file directory.

C.) link to the third handle (via USB connection computer)

Here is an example of the file externalcamera. CFG:

x=0 
       y=0 
       z=0 
       rx=0 
       ry=0 
       rz=0 
       fov=60 
       near=0.1 
       far=100 
       sceneResolutionScale=0.5 

When all three conditions are established, the content program will set up a virtual camera in the scene, and the file externalcamera. CFG is the relevant parameter that defines the virtual camera (see figure 3) :

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


. X, y, z (unit: meter) : the 3d distance between the virtual camera and the third handle.

. Rx, ry, rz (unit: degree) : the virtual camera is opposite to the third handle.

. Fov: the vertical fov of the virtual camera. (this FOV should be the same as the entity camera's FOV)

In addition to defining the parameters of the virtual camera, this file also defines some screen parameters:

. Far (unit: meter) : background screen shows the farthest distance. If the game scene is very large, it is suggested to adjust this parameter.

. SceneResolutionScale: the quality of the game screen, reducing the quality of the picture can reduce the use of computer resources.

How to calibrate camera parameters (calculate vertical FOV)

When shooting a mixed reality film, the most important thing is to get the FOV parameter value and the distance between the third handle and the physical camera to successfully synthesize the foreground and the entity object. Here we use AR's tools to calculate the features of the camera and lens, and get the correct vertical FOV (unless the camera or lens is changed at the time of shooting), and this step can be performed once.

1.) download ARToolKit for Unity (http://artoolkit.org/dist/arunity5/5.3/ARUnity5-5.3.2-tools-win.zip)

2.) in ARToolKit for Unity is found in the Calibration model "Calibration chessboard (A4). PDF" and in accordance with the original size print (path: [downloaded ARToolKit root directory] / doc/patterns)

3.) from the command prompt characters executable program calib_camera. Exe (path: [downloaded ARToolKit root directory] / bin)

The instructions are as follows:

> calib_camera. Exe -vconf "-devNum=1 - flipv-showdialog"

The calibration tool must obtain the picture of the solid camera. If the front lens is captured, please modify the instruction parameter to 1 or 2.

Step 4.) according to the online teaching (http://artoolkit.org/documentation/doku.php? The id=2_Configuration:config_camera_calibration produces the camera's characteristic parameter file, camera_para.bytes (see figure 4)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


"If the result of calibration is applicable, the error of each image should be within one pixel. If the error exceeds two pixels, it indicates that the calibration result is not successful, please recalienable."

5.) the calibration will be produced after the file camera_para. Bytes is copied to the displacement calculation tool externalcamera_cfg_gen.

$PATH \ externalcamera_cfg_gen \

Externalcamera_cfg_gen_Data \

StreamingAssets \

Ardata \

Fix the third handle on the camera.

Next, we tried to calculate the distance between the third handle and the physical camera. It is the ease of calculation depends on the will handle is fixed on the way of the camera, and the relative position, they need to modify the file externalcamera. The CFG, therefore suggested to find a fixed and stable way, such as the handle on the hot shoe, remove the installation fast, and the relative position easily fixed (see figure 5)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


In order to render the handle to the same level as the physical camera, we set up a handle bracket, which is vertically mounted on the camera, and the triaxial rotation in the file externalcamera. CFG is very close to zero. You can download 3D printing files and paste one to two hexagon nuts (including 1/4 thread) after printing (please refer to figure 6)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


( 3D Print file download link:https://drive.google.com/file/d/0B9XEEDfLPxmjTkc2RF85OVo4clk/view?pref=2&pli=1 ) 

 Once the third handle is successfully fixed to the foot frame, camera bracket or hot boot, the next step is to calculate the desired displacement (x, y, z) and rotation (rz, ry, rz) in the file externalcamera. We suggest the following two ways:

Solution A:

If you use HTC Vive handle bracket and should be fixed on the camera's hot shoe, the handle and the camera lens should be in the same level, all the relative rotation (rx and ry, rz) is close to zero, so we only need to calculate displacement:

1.) find the center of the handle (please refer to figure 7) :

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


The center of the handle is located on the upper edge of the disc tracker. In the figure below, the position of the rectangle represents the center point of the handle.

2.) to calculate the displacement (x, y, z) : through the tape measure tools such as calculate the center of the handle and the position of camera lens imaging, three dimensional distance above the lower right is the x, y, z axis and the relative position of handle.

Solution B:

If you grip and the camera is not in the same level, here provide a displacement calculation tool (download link: https://drive.google.com/file/d/0B9XEEDfLPxmjZjhYaEZrNE9mLVE/view)

1.) download ARToolKit for Unity (http://www.artoolkit.org/dist/arunity5/5.3/ARUnity5-5.3.2-tools-win.zip)

2.) in the ARToolKit for Unity, find the calibration sample "Multi pattern 4x3 (A4). PDF" and print according to the original size. (path: [downloaded ARToolKit root directory] / doc/patterns)

3.) the calibration will be produced after the file camera_para.bytes is copied to the displacement calculation tool externalcamera_cfg_gen.

$PATH \ externalcamera_cfg_gen \

Externalcamera_cfg_gen_Data \

StreamingAssets \

Ardata \

4. Link the camera to the computer.

5.) $PATH \ externalcamera_cfg_gen \ externalcamera_cfg_gen exe

. Please confirm that the calculation tool captures the correct calibration parameter filename, which you can specify in the file videopara. CFG: videoCParamName0 = camera_para.

. The calibration tool must obtain the picture of the solid camera. If the front lens of the helmet is captured, please modify the parameters in the file videopara. CFG to 1 or 2:

VideoConfigurationWindows0 = - devNum = 1 - showDialog - flipV

. Please confirm that the screen ratio of the physical camera is 16:9, for example: 1920 x 1080.

6.) wearing the top

7. Place the two handles in the position of the blue virtual handle (please refer to figure 8)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


8.) remove the head

9.) calibrate the sample "Multi pattern 4x3 (A4). PDF" as for the center of the two handles, the direction should be consistent with the virtual scene (please refer to figure 9)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


10. The camera is aimed at the calibration sample "Multi pattern 4x3 (A4). PDF". (please refer to figure 10)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


11. wear the top.

12. Press the Grip button on the side of the next handle, and switch the virtual orange handle to the position of the third handle. The relative position of the orange handle and the virtual camera is the same as the physical condition (please refer to figure 11)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


13. Press the Trigger button of the next handle to produce the file externalcamera000.cfg.

14.) to rename the file to externalcamera. The CFG and copy it to the game executable file folder this tool also provides "floor" correction function, when you find that there are slightly sloping floor (handle can't accord with the blue handle), may affect the calculation results, please through the calibration function first floor height.

Image through OBS Studio.

Next, we will through the OBS Studio software to synthetic images, before you start, need to decide the quality of the output film, and the quality depends on your object and purpose, the following is the case of setting, make adjustment when necessary:

. Resolution: 1080P, 30FPS.

. Film size: 100~200MB (3-5 minutes)

1.) set > output > volume "video" in OBS Studio > (please refer to figure 12) :

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


. Output mode: advanced.

. Type: standard

. Video format: mp4.

. Traffic: 6000

2. Set > image in OBS Studio > (please refer to figure 13) :

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


. Source (full screen) resolution: 1920x1080.

. Output (zoom) resolution: 1920x1080.

. Commonly used FPS: 29.97.

3. In order to obtain a video with 1080P resolution, the game must be opened on 4K screen with full screen (please refer to figure 14):

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


. Press Shift to execute the game execution file (.exe)

. Cancel "Windowed"

. Screen resolution: 3840 x 2160.

4. In OBS Studio, three new sources must be added in the following order: "foreground", "image acquisition" and "background" (please refer to figure 15) : 

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


A.) prospect setting (please refer to figure 16)

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


The foreground screen shows the visual Angle of the virtual camera. In the game scene, all objects between the head and the virtual camera are displayed in this picture. Because foreground objects need to be stacked above the player, the foreground layer needs to be uppermost.

. Add a new source "access window"

. Window: specifies the name of the execution file for the game.

. Cancel "capture cursor"

Add two special effects filters in the foreground layer:

. Crop (right: 1920, bottom: 1080)

. Color Key (Black) handles the Black part of the foreground layer.

B.) image acquisition setting [resolution: 1920x1080] (see figure 17) 

 

如何透过HTC Vive拍摄Mixed Reality (混合现实)影片


This source is mainly collected from the camera image, so a new image capture device is added:

. Device: select network camera or video acquisition card.

. Resolution /FPS type: custom.

. Resolution: 1920 x 1080.

Add a special effect filter:

. The Chroma Key (Green) handles the Green screen behind the player, and can adjust the "Similarity" and "Smoothness" to achieve a better back effect.

C.) background setting

The background screen shows the visual Angle of the virtual camera. In the game scene, all objects that are in the background of the head and distant background will be displayed in this picture.

Add a special effect filter in the background layer:

· Crop (right: 1920, upper: 1080) 


Related news / News More
  • Clicks: 71
    2017 - 11 - 09
    As is known to all, all rendering work of Unity3D is inseparable from the Shader, namely Shader. It is actually a small program that is responsible for combining the input Mesh in a specified manner with the input map or color, and then the output. The drawing unit can draw the image to the screen based on this output. The input map or color, plus the corresponding Shader, and the specific parameter Settings for Shader, will be packaged and stored together, resulting in a Material. After that, we can render the material to the appropriate renderer. In the end, Shader is a program that specifies the points and colors that can be read by the input color map and the output renderer.1. Type of Shader.Shader generally falls into three categories:1. Surface Shader -- Surface Shader.Contains illu...
  • Clicks: 54
    2017 - 12 - 08
    The main interfaceInterface in some function is not perfect, but I'm still posted the latest interface, open to you is to want to tell you that each person has a different tool interface, have their own tools, so I will share here Interface source reference: Transform relative Transform position, direction, size.  Refer to the above script, for example setting location, direction and size. The advantage of using it is that it can ignore the changes caused by the object.Fast switch object activation close.  Shortcut Ctrl+1 to quickly activate or close the selected object. Clicking on the upper-left corner of the property panel with the mouse will be tricky.Fast marshalling objects. Ctrl+4, which adds a parent node to several selected objects, is equivalent...
  • Clicks: 79
    2018 - 01 - 12
    1: pull a "Cube" in the scene and attach the camera and the moving object to the Cube.2: create a lens.3: click on the animation to create a "Matinee".4: select the shot to create a "New Camera Group" on the left side, and set the time for animation on the right time axis.5: select "Movement" to press enter and a keyframe will appear on the right.6: go back to the scene and drag the camera to set the track.7: select the Cube you just created, create a New Empty Group on the right, and right-click to add the New "Movement Track".8: mouse the left key to select key frames 1 press the return key, press and hold Ctrl can appear not to drag another key frames and drag it to the appropriate location, you can create multiple keyframes, create good...
  • Clicks: 138
    2018 - 02 - 02
    Maybe you're a developer who wants to make cool commercials for your HTC Vive game. Or you are the game anchor, want to bring the high quality VR live experience for the audience; Even you are a fan of VR. Anyway, if you're interested in MixedReality, go ahead. This article describes how to use HTC Vive to shoot high quality MR video.What is Mixed Reality(Mixed Reality) film?Mixed Reality film is a film that is synthesized through the images in VR. Mixed Reality video is more suitable for live-streaming and propaganda films than the VR films recorded in general methods.If it is still not clear, you can see the following film, which is the film effect produced by Mixed Reality.http://v.qq.com/x/page/x0194621p8b.htmlMake the props for the Mixed Reality film.1.) green screen environment2.) ca...
  • Name
  • *
  • company name:
  • *
  • Address:
  • *
  • Phone
  • *
  • fax:
  • *
  • E-mail:
  • *
  • Postal code:
  • *
  • Real estate name
  • *
  • Content
  • *
     
Room 301, Bldg 1, No.388 Zhongjiang Road, Shanghai
64683109
18665895116
sales@vrwaibao.com
Follow us
Mobile phone side
WeChat public number
Online message
Contact us
Copyright ©2017 - 2020 Zhi Jing Technology