Results of the Kaleidescape vs Streaming Fall Mini-Event

If you’d like to discuss the results of this event or are interested in any of these products, please contact us at info@thescreeningroomav.com or call us at 720-377-3877. Content like this is difficult and expensive to produce; the business we hope to earn from this is what funds the next big event. If you find this kind of content valuable, please reach out to us about any of your audio / video needs so we can keep putting out this kind of good, reliable information. We'll be hosting a webinar to discuss in detail what we learned from the event. If you’re interested in attending the webinar, be sure to sign up for our mailing list.

INTRODUCTION

So, we finally did our basic, down and dirty blind comparison between streaming and Kaleidescape / physical media. This test was primarily focused on testing the claim that streaming audio is "TERRIBLE" when compared to physical media or Kaleidescape (though video differences were also noted during our testing). We did our best to level-match the audio from the various sources so that they played at the same volume level in the room.

At no time did the viewers / listeners know which source was playing at any given time, except during our disc vs. Netflix test later in the evening.

Since we only held a two-hour event, this was primarily a "proof of concept" to see if streaming and physical could even be properly compared. There are some major difficulties comparing the formats:

• On-screen displays can give away the source

• Streaming needs a few seconds to get up to full quality 

• There is no way to precisely level match content, as there are no test tones on streaming. For this reason, we spent several hours getting average SPLs for the scenes compared, and then dialing in the volume offset in our Trinnov-derived JBL SDP-75 processor.

Testing was supervised by our good friends Kris Deering of Deep Dive AV and Sammie Prescott Jr. of Projector Central. Matt Scarborough of Kaleidescape was also in attendance to answer questions and brought a Kaleidescape Strato V Movie Player for demonstration on our Sony 77-inch BRAVIA XR A95L OLED.

Steve Crabb and I came up with a way to blind the content using a separate hidden monitor where Steve could cue up the scenes and make sure the picture and sound had stabilized before we passed the picture on to our JVC DLA-RS4200 projector (using the "hide" function on the JVC).

Reference playback system including JVC DLA-RS4200, Panamorph Alpha lens, 144-inch Stewart Filmscreen, JBL Synthesis Trinnov processor, Revel PerformaBe speakers, Seaton Sound subs, and Kaleidescape Strato/Terra system.

Source Components

Kaleidescape Strato C Player and Kaleidescape Terra 48TB Server

Oppo UDP-203

AppleTV 4K X

Streaming Services

ITunes

Disney +

Amazon Prime

Paramount +

Netflix 

Playback System Components

Trinnov Altitude32 (JBL SDP-75 derivative)

Revel PerformaBe 9.6.6 speaker system (winner of our blind speaker tests)

Six Seaton Sound Custom Install subs, calibrated by Mark Seaton himself (amazing)

JVC DLA-RS4200 projector

144" Stewart Filmscreen StudioTek130 Scope Screen

Test Material

Top Gun Maverick (opening sequence)

The Greatest Showman (opening sequence - "The Greatest Show")

Godzilla Minus One (opening sequence, boat chase)

Fleetwood Mac - Rumours - The Chain (Atmos Music)

We had about 15 people in the room (and asked them to switch seating positions several times).

Level Matching

AN IMPORTANT NOTE ABOUT THE TEST (for those who want to try to recreate this at home):

LEVEL MATCHING is absolutely critical - but difficult to do. 

These are the big volume offsets we had to use to get the levels between sources to match in the room. Values are what the volume level was showing on the Trinnov. Note that -20 is roughly "reference level" on the Trinnov volume scale:

Maverick 

Apple -10

Prime -16

Paramount -12

Kscape -22

Greatest Showman 

Apple -10

Disney+ -10

Kscape -18

Godzilla Minus One 

Disc -20

Streaming -10

Fleetwood Mac

Disc -22

Streaming -20

The UMIK-1 set up for level-matching between streaming sources and Kaleidescape.

What does all that mean? Well, for MAVERICK, we had to "turn up" the iTunes source 12 dB so that it was the same level as Kaleidescape. That's HUGE! Listening tests reveal that differences as small as 1 to 2 dB can significantly impact your perception of the quality of sound. And in the case of iTunes, there was a massive 12 dB difference to compensate for. Amazon Prime had a -6 dB difference for the same title, so again, level matching had to be done.

So, if you're trying to conduct a legitimate comparison, you need to make sure levels are precisely matched. 

On the positive side, getting the levels to match ultimately just means you have to turn up the volume, generally on the streaming source.

Dialnorm

A quick note on Dialnorm to contextualize the aforementioned volume issue. Dialnorm is a bit of metadata embedded in streamed content. It's essentially a gain setting to tell an AVR or processor to play back a particular piece of content with a certain level. It's embedded in the content by the content's author and is very similar to "second volume knob". It's the reason for when you switch from a disc to streaming and have to turn it up louder to hear the same volume level in the room. Because it's like a "second volume setting", there's technically nothing wrong; as was previously mentioned, you just need to turn it up louder.

Streaming vs Kaleidescape

Our test was designed to show streaming at its very best. However, there are lots of variables can influence the quality of streaming, as listed below:

• You don't always know what audio format is being pushed out - sometimes it's just stereo. Maverick on Amazon was only 5.1 for example, but Atmos on the other services tested with no clear indication that this is so. Sometimes streaming will default to plain old stereo.

• Different streaming services have different configuration settings that need to be set up properly in the app itself.

• Different streaming devices need to have their audio output settings set up properly (such as bitstream out).

• Results may vary depending on which streaming device you use since some don’t support the same features as others on an app-to-app basis (including Atmos, frame-rate matching, and sometimes even HDR).

• Different tiers of pricing to get the best sound and picture formats, and sometimes it's very confusing to determine what you're even paying for or getting.

• Which movie is on which service, and how does the quality between the services impact the quality that you get (for example, Amazon Prime was worst in our tests for audio and video).

• Internet speed can be a big variable. Our test had gigabit speeds so streaming was at its best, which is not always the case - particularly if you are on Wi-Fi (we had a direct network connection).

• Streaming can vary in quality moment to moment.

• Movies get pulled from services often without warning.

• Streaming can also be affected by how many people are using the internet in your house or neighborhood, and how much your ISP might be throttling your service.

KALEIDESCAPE ELIMINATES ALL THOSE VARIABLES. Disc does as well, but not if the disc gets scratched or starts glitching. And of course Kaleidescape is almost as convenient as streaming - you decide what you want to watch and 20 minutes or so later it's ready at the very best quality without any of the aforementioned issues.

Event attendees comparing streaming content to Kaleidescape.

THE RESULTS

Differences in video quality were noticed by most attending, though they often weren't "huge." However, since this was audio-focused, we did not really put any torture test material up. Will be interesting to do a video stress test as a follow-up.

Top Gun Maverick

Amazon had noticeably poorer video than Paramount or iTunes for Maverick, and the mix was only 5.1 (vs. Atmos on the other platforms). These differences were noted by almost everyone present.

Amazon streaming pretty much stinks. Apple was reasonable as was Disney I thought. In one test I clearly found KScape audio significantly better. In one test I did not. I saw no difference in video services except for Amazon. - Todd J, Event Participant

I thought 3 (Kaleidescape) was the best, but I thought 2 (Amazon) was noticeably inferior. - James H, Event Participant

I was watching a 12-inch 1080p display while I was controlling the show, and I noticed that there was more detail in the Kaleidescape picture in Top Gun Maverick. Where it easily visible was in the wide shots with the aircraft carrier and bridge silhouetted into the sky. The railings and antennas clearly showed more detail. - Steve C, TSR AV

Several people noticed distinctly sharper images with Kaleidescape, particularly against Amazon Prime. However, it's important to note that that was for Maverick only, as that's the only Amazon title we tested. But most people noticed that the Amazon picture was not up to the quality of Apple or Kaleidescape, plus almost everyone noticed the lack of immersion due to the fact that Amazon only had a 5.1 mix.

The Greatest Showman

Greatest Showman was the most interesting test, as most people there thought it was a toss-up audio-wise between Apple, Disney+ and Kaleidescape. In fact, two people said that the iTunes stream seemed a tiny bit punchier and dug a little deeper. However, it is our considered opinion that this could be a simply a difference in level, as there was no way to precisely match levels between sources. The Kaleidescape is lossless after all, so is reference-quality. Still, it was interesting that most people there could not reliably tell which was which on this test audio-wise - there was no clear preference expressed.

This absolutely demonstrates the importance of level-matching, as without level-matching, Kaleidescape was 8 dB louder than both Apple and Disney+. Once that 8 dB difference was accounted for, there was no consensus as to which sounded "best." Note that several people noticed better picture quality with Kaleidescape, with an uptick in sharpness.

The first one (Disney), I thought the audio was "dull". It seemed like the colors were also less vibrant. Visually, I thought the second one (Kaleidescape) was best; more vibrant, contrast, and colors. - Doug K, Event Participant

I thought 1 (Disney) was missing the heights. - Event Participant

1 (Disney) sounded the worst, but I couldn't tell much difference in the video. - Event Participant

On the third one (Apple), the low-frequency seemed to be much more defined and much more prevalent and the detail seemed to be better, too. - Event Participant

I felt like the vocals were better on the third one (Apple). - Event Participant

Godzilla Minus One

When we compared the UHD disc of Godzilla Minus One played from the Oppo to Netflix on Apple TV 4K, it was quite obvious that the disc was sharper. When switching audio, again - big differences came down to the mix chosen. Netflix defaults to the 5.1 English dub track, which is a whole different mix with neutered surround activity and dynamics (in certain scenes). However, the Japanese Atmos mix on Netflix sounded very similar to the Atmos mix on the disc. This test came later in the evening so was not as structured (and was not blind).

CONCLUSIONS

So, what we have here is a limited but valid controlled listening test between streaming on several popular services, using the content that got the most votes in an online Facebook poll where we asked home theater enthusiasts which titles sounded "obviously and dramatically better" on Kaleidescape / disc vs. streaming. The top vote getters were Top Gun Maverick and The Greatest Showman.

When we say this test was limited, we mean that we did not conduct multiple listening trials where content was randomized, nor did we look for subtle differences in sound quality that might have been revealed if we had done more focused and repeated sessions that highlighted potential problematic sequences. It is possible that this kind of testing might have revealed the more subtle differences one would expect to hear with lossy audio. Further, if we had done listening tests comparing the various free or lower-tier streaming services like Pluto, Freevee and Tubi, we would almost certainly have heard rather large differences as those services are frequently stereo-only.

One of our conclusions is that many people do not really understand how lossy audio compression works. Lossy compression works very hard to throw away content that is inaudible, by means of a process called "perceptual coding" (though audible artifacts can start showing up if the data rate gets low enough). It does NOT change dynamics or bass response. Dynamic compression, on the other hand, CAN affect dynamics. We didn't hear any evidence of dynamic compression at work in the material we chose - the dynamics were essentially the same between all sources.

Another thing we thought we might hear was filtering of bass frequencies. We did not hear this either in any of this content - bass and dynamics were roughly the same once level matching was done. However, it's always important to point out that we only tested a tiny percentage of the content that's out there. Certainly there could be filtering or dynamic compression happening with other titles or services. In fact, it wouldn't surprise us at all.

All of that said, based on this first round of testing, those who claim to hear "huge," "night and day," and "dramatic" differences between the streaming audio off a high-quality streaming source and that of disc or Kaleidescape would be greatly served by simply adjusting their volume knob when streaming services are the source. Sometimes there was up to a 12 dB difference in level, which means quite a large turn of the knob. Even in that type of situation, we found that dynamics and fidelity were generally retained with top-tier streaming services vs. their physical counterparts.

While this test is hardly definitive, I think we demonstrated that the big differences most people claim to hear can be mitigated by simply turning up the volume on streaming sources, and choosing a good streaming provider like Apple, Paramount, or Disney (and making sure you are paying for their highest-quality service). The choice of streaming device can matter here too, as there are differences in how various devices output HDR, frame rate, and audio format (which can also vary from app to app).

"We've all experienced issues with streaming where the dreaded spinning circle shows up on our screen, or where the audio is just stereo, or where the picture suddenly looks worse than VHS before ramping up to DVD quality than back down to VHS again. I hardly ever stream and I still am frustrated by those issues." - John Schuermann, Owner of TSR AV

One final point. None of this is intended to imply that streaming is as good as Kaleidescape (or disc). While there are good reasons to use streaming (exclusive content being the main one), we think it's also clear that there are enough variables that streaming is still often problematic for many - even if much of the time it's almost as good. Kaleidescape is far superior to streaming, though. With better video and audio quality, permanent ownership, playlists, instant access to scenes, no buffering or internet/network issues, and high-quality hardware, it's still the very best way to know you're going to experience the highest-quality content in your home theater.

For content I care about, it's disc or Kaleidescape. For casual viewing, streaming is not as bad as some lead you to believe. I think of streaming as the sampler, like going to a buffet. But for a fine meal, I go to a real restaurant. - John Schuermann, Owner of TSR AV

NOTE - If you’d like to discuss the results of this event or are interested in any of these products, please contact us at info@thescreeningroomav.com or call us at 720-377-3877. Also, we'll be hosting a webinar soon to present and discuss in detail what we learned from the event. If you’re interested in attending the webinar, be sure to sign up for our mailing list.