All Things 3D | Test of 8K (4K Per Eye from 6K) "Immersive Spatial Video" (iSpatial Video) with AmbiX Audio @allthings3d | Uploaded 4 months ago | Updated 3 hours ago
Over the course of six months I have been working on a stereo camera system that provides the resolution and fidelity needed for current & future recent VR/XR headsets, in particular the @Apple #visionpro . With the AVP's 3660 horizontal resolution for approximately 100-110 degrees per eye, I figured you would need a hemispherical resolution of 5.7K - 6.3K per eye. I chose 6144x6144 as a multiple of 1024, which allows me to easily scale to 4K for Quest 3 and here as a 8K (4K per eye) #VR180 on YouTube.
To create such a high resolution 180 video, you need a high resolution camera that not only provides at least 6K, but also at least 10 bit depth per RGB color and fairly large sensor to minimize noise in low light situations. Not only is such a camera kind of a "pie in the sky" endeavour, the price makes it almost impossible to achieve. Canon has come the closest with its $4K 'Creator' edition combining an EOS-5C and dual fisheye lens bifurcating the 8K sensor into two, 210 degree FOV images that can be combined to create an 'iSpatial Video', for an even more expensive & complicated solution; you can combine larger cameras from RED and Black Magic Industry using partial mirror technologie, but in no way are these compact or portable, but you can achieve 6K and even 8K. There are other VR180 cameras available but most max out at 3K per eye, and at the high end they are still thousands of dollars in cost. So you may ask, how can one create 'Immersive Spatial Video' without breaking the bank?
Enter DJI and GoPro. The former has several cameras that can support 5K in 4:3 but there seems to be only one camera with the form factor that allows 4:3 to be used with two of these side by side at the standard 63mm-65mm interocular spacing and that is the DJI OSMO Action 2 - dual camera system I experimented allowing 156 diagonal FOV (spherical FOV) at a 4:3 ratio at 5120 60 fps. You might be saying -- wait this isn't 180 degree FOV and you are right. However, I have found that even 155-165 degrees is more than enough to extend past the 100-110 FOV in VR headsets to achieve an immersive experience. In fact, I have found that anything past 130 degree FOV, loses much of the 3D stereo effect anyway. However the Osmo Action 2 has a glaring deficiency and what I feel is important requirement if you want realistic videos -- and that is 10-bit dynamic range. Sadly, this is not possible with the Action 2, limited to a webcam like 8-bit dynamic range. So my search continues and I came upon the GoPro Hero Black 11 Mini. Even though this is not GoPro's latest Hero Black 12, it shares the same Sony image sensor, lens, in fact almost all the same features as the 12, except two which I will discuss soon. First the Mini 11 is smaller than the 11 or 12, but lacks the screens to keep the size smaller. Small enough to mount two side by side, as close as 60 mm, but at 63mm, there is still enough of a gap to allow a right-angle USB-C female adapter to be installed, allowing the charging or data retrieval without having to remove it from your stereo camera bracket. Also without a back screen, GoPro added heatsink keeping it cooler & no screens allows it to draw less power. However, from a power perspective it does not have a removable battery, limiting your recording time to about 40 minutes without external power. Plus the Mini 11 like many of the GoPro cameras can use the 'Lab' firmware allowing you to use QR codes to instantly change camera settings and other "PRO" features not available in the standard firmware, like higher bitrate settings from the current 'High' setting of 120 bps up to 200 bps allowing for higher quality HEVC files (sadly no uncompressed mode) , flat and LOG mode, shutter angle and disabling proxy (LRV) and WAV files to reduce processor load -- concentrating ONLY on the main file. However you are not blind, the camera(s) can be linked to the iOS or Android app to allow preview & standard/protune settings to still be made after your QR code initial settings. Plus with a low-cost BT multicam remote, you can start recording near simultaneously (less 1 frame). All this at 8:7 5312x4648 resolution, 158 FOV, 10bit, Log, 200bps, variable shutter angle and ISO/shutter-rate. What's the catch??? Only 30fps. This isn't such a bad problem if your cameras are stationary and you use a shutter rate of at higher than 1/60, which is easy to do outside where you shutter rate can easily reach 1/480 at 100 ISO. (I normally set mine 1/120 - 100 ISO locked) & use a 1 stop circular polarizer.
However, iSpatial Video capturing with either of these two systems is not for the faint of heart & complete "how-to" workflow article on Medium/Substack coming soon, with links to project files, settings & LUTs for DaVinci Resolve Studio, Dolby ATMOS & FFMPEG.
8K SBS for PCVR or Quest 3 - (mega.nz/folder/wvITkB4b#DhAWuiJUjphf3-PKtn4wEg)
12K iSpatial Apple Vision Pro - (mega.nz/folder/06RyDYzB#3af6wbWgKc0apY_vih20Vg)
Over the course of six months I have been working on a stereo camera system that provides the resolution and fidelity needed for current & future recent VR/XR headsets, in particular the @Apple #visionpro . With the AVP's 3660 horizontal resolution for approximately 100-110 degrees per eye, I figured you would need a hemispherical resolution of 5.7K - 6.3K per eye. I chose 6144x6144 as a multiple of 1024, which allows me to easily scale to 4K for Quest 3 and here as a 8K (4K per eye) #VR180 on YouTube.
To create such a high resolution 180 video, you need a high resolution camera that not only provides at least 6K, but also at least 10 bit depth per RGB color and fairly large sensor to minimize noise in low light situations. Not only is such a camera kind of a "pie in the sky" endeavour, the price makes it almost impossible to achieve. Canon has come the closest with its $4K 'Creator' edition combining an EOS-5C and dual fisheye lens bifurcating the 8K sensor into two, 210 degree FOV images that can be combined to create an 'iSpatial Video', for an even more expensive & complicated solution; you can combine larger cameras from RED and Black Magic Industry using partial mirror technologie, but in no way are these compact or portable, but you can achieve 6K and even 8K. There are other VR180 cameras available but most max out at 3K per eye, and at the high end they are still thousands of dollars in cost. So you may ask, how can one create 'Immersive Spatial Video' without breaking the bank?
Enter DJI and GoPro. The former has several cameras that can support 5K in 4:3 but there seems to be only one camera with the form factor that allows 4:3 to be used with two of these side by side at the standard 63mm-65mm interocular spacing and that is the DJI OSMO Action 2 - dual camera system I experimented allowing 156 diagonal FOV (spherical FOV) at a 4:3 ratio at 5120 60 fps. You might be saying -- wait this isn't 180 degree FOV and you are right. However, I have found that even 155-165 degrees is more than enough to extend past the 100-110 FOV in VR headsets to achieve an immersive experience. In fact, I have found that anything past 130 degree FOV, loses much of the 3D stereo effect anyway. However the Osmo Action 2 has a glaring deficiency and what I feel is important requirement if you want realistic videos -- and that is 10-bit dynamic range. Sadly, this is not possible with the Action 2, limited to a webcam like 8-bit dynamic range. So my search continues and I came upon the GoPro Hero Black 11 Mini. Even though this is not GoPro's latest Hero Black 12, it shares the same Sony image sensor, lens, in fact almost all the same features as the 12, except two which I will discuss soon. First the Mini 11 is smaller than the 11 or 12, but lacks the screens to keep the size smaller. Small enough to mount two side by side, as close as 60 mm, but at 63mm, there is still enough of a gap to allow a right-angle USB-C female adapter to be installed, allowing the charging or data retrieval without having to remove it from your stereo camera bracket. Also without a back screen, GoPro added heatsink keeping it cooler & no screens allows it to draw less power. However, from a power perspective it does not have a removable battery, limiting your recording time to about 40 minutes without external power. Plus the Mini 11 like many of the GoPro cameras can use the 'Lab' firmware allowing you to use QR codes to instantly change camera settings and other "PRO" features not available in the standard firmware, like higher bitrate settings from the current 'High' setting of 120 bps up to 200 bps allowing for higher quality HEVC files (sadly no uncompressed mode) , flat and LOG mode, shutter angle and disabling proxy (LRV) and WAV files to reduce processor load -- concentrating ONLY on the main file. However you are not blind, the camera(s) can be linked to the iOS or Android app to allow preview & standard/protune settings to still be made after your QR code initial settings. Plus with a low-cost BT multicam remote, you can start recording near simultaneously (less 1 frame). All this at 8:7 5312x4648 resolution, 158 FOV, 10bit, Log, 200bps, variable shutter angle and ISO/shutter-rate. What's the catch??? Only 30fps. This isn't such a bad problem if your cameras are stationary and you use a shutter rate of at higher than 1/60, which is easy to do outside where you shutter rate can easily reach 1/480 at 100 ISO. (I normally set mine 1/120 - 100 ISO locked) & use a 1 stop circular polarizer.
However, iSpatial Video capturing with either of these two systems is not for the faint of heart & complete "how-to" workflow article on Medium/Substack coming soon, with links to project files, settings & LUTs for DaVinci Resolve Studio, Dolby ATMOS & FFMPEG.
8K SBS for PCVR or Quest 3 - (mega.nz/folder/wvITkB4b#DhAWuiJUjphf3-PKtn4wEg)
12K iSpatial Apple Vision Pro - (mega.nz/folder/06RyDYzB#3af6wbWgKc0apY_vih20Vg)
