Streaming video from Luwak to browsers not working so well

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Streaming video from Luwak to browsers not working so well

John Axel Eriksson
First a little background:

We've been putting videos in luwak without much trouble for quite a while now, small and a little bigger (by bigger I mean around 6 or 7 megs).

To stream these so far, we've basically done this:

Get the web app to download the video over http from luwak and store on local disk.
Get nginx, through an x-accel, to actually send it to the client (i.e the browser).

For various reasons this isn't really how we want to do it. There is one big reason why we do this currently. We need all clients to authenticate through
our rails-app. So we first tried to have rails send an x-accel to nginx which would then proxy to luwak. This won't work because nginx doesn't support
http 1.1, chunked encoding and all that jazz to backends at least (it's 2011… why I just don't know).

Anyway, for a while now we've thrown node.js into the mix(for other reasons) and recently I started working on some code for node.js to recognize
x-sendfile (similar to x-accel). I also made it do the same type of proxying as nginx can do - but this time it does http 1.1. Unfortunately I couldn't get it
to work. So I started investigating riak/luwak directly. Could we even stream a video out of luwak directly? No we couldn't. So the problem goes all the
way back to riak.

Using Safari it won't stream more than a few seconds and then sound and video stops (though the progress bar keeps moving).
Using Chrome and Firefox I could stream perhaps half the video before more or less the same thing happened.

I then cloned riak and removed one line in deps/luwak/src/luwak_wm_file.erl, specifically line 363 -      {"gzip", fun(X) -> zlib:gzip(X) end}]. So that the list
now only contains [{"identity", fun(X) -> X end}]. (Those are the default_encodings). I will admit I didn't know exactly what I was doing.

Built riak and did some more testing. The results were interesting. Suddenly Chrome and Firefox would play the whole file through (a .webm).
Unfortunately Safari still did the same thing, i.e played 4 seconds and then stopped video and sound (while still showing progress).

I didn't try IE and quite frankly I'm afraid that one might just blow up my computer…

Anyway, I just wanted to ask the list if anyone ever tried the same thing or if any of the good people at basho knows more than I do (which I suspect
they do :-).


Thanks,
John
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Streaming video from Luwak to browsers not working so well

Sean Cribbs-2
John,

If the browser negotiated a gzip response, the results when streaming would be incorrect (or at least incompatible) if the response were ever larger than a single chunk. Webmachine would send each chunk through the gzip compressor individually, whereas browsers would expect the entire response to be a single gzip blob (with a single header/checksum). At least, that's why the gzip thing didn't work.  Not sure why Safari broke even when you turned that off.

On Tue, Sep 13, 2011 at 4:04 PM, John Axel Eriksson <[hidden email]> wrote:
First a little background:

We've been putting videos in luwak without much trouble for quite a while now, small and a little bigger (by bigger I mean around 6 or 7 megs).

To stream these so far, we've basically done this:

Get the web app to download the video over http from luwak and store on local disk.
Get nginx, through an x-accel, to actually send it to the client (i.e the browser).

For various reasons this isn't really how we want to do it. There is one big reason why we do this currently. We need all clients to authenticate through
our rails-app. So we first tried to have rails send an x-accel to nginx which would then proxy to luwak. This won't work because nginx doesn't support
http 1.1, chunked encoding and all that jazz to backends at least (it's 2011… why I just don't know).

Anyway, for a while now we've thrown node.js into the mix(for other reasons) and recently I started working on some code for node.js to recognize
x-sendfile (similar to x-accel). I also made it do the same type of proxying as nginx can do - but this time it does http 1.1. Unfortunately I couldn't get it
to work. So I started investigating riak/luwak directly. Could we even stream a video out of luwak directly? No we couldn't. So the problem goes all the
way back to riak.

Using Safari it won't stream more than a few seconds and then sound and video stops (though the progress bar keeps moving).
Using Chrome and Firefox I could stream perhaps half the video before more or less the same thing happened.

I then cloned riak and removed one line in deps/luwak/src/luwak_wm_file.erl, specifically line 363 -      {"gzip", fun(X) -> zlib:gzip(X) end}]. So that the list
now only contains [{"identity", fun(X) -> X end}]. (Those are the default_encodings). I will admit I didn't know exactly what I was doing.

Built riak and did some more testing. The results were interesting. Suddenly Chrome and Firefox would play the whole file through (a .webm).
Unfortunately Safari still did the same thing, i.e played 4 seconds and then stopped video and sound (while still showing progress).

I didn't try IE and quite frankly I'm afraid that one might just blow up my computer…

Anyway, I just wanted to ask the list if anyone ever tried the same thing or if any of the good people at basho knows more than I do (which I suspect
they do :-).


Thanks,
John
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



--
Sean Cribbs <[hidden email]>
Developer Advocate
Basho Technologies, Inc.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Streaming video from Luwak to browsers not working so well

John Axel Eriksson
I have no idea why Safari won't work unfortunately. Perhaps it's the type of range-requests it's doing - I know Safari is a bit special in this regard (as are the iDevices). Is there at least a way, other than changing the erlang code and recompiling, to get luwak to not gzip the response? Seems that the browsers ask for it even though it won't work. Although the browsers may be stupid - that's what I have to work with. Another idea I had was that perhaps I could change the headers when proxying using node.js.

(sorry for the dupe Sean - that got sent only to you).

On Tue, Sep 13, 2011 at 10:40 PM, Sean Cribbs <[hidden email]> wrote:
John,

If the browser negotiated a gzip response, the results when streaming would be incorrect (or at least incompatible) if the response were ever larger than a single chunk. Webmachine would send each chunk through the gzip compressor individually, whereas browsers would expect the entire response to be a single gzip blob (with a single header/checksum). At least, that's why the gzip thing didn't work.  Not sure why Safari broke even when you turned that off.

On Tue, Sep 13, 2011 at 4:04 PM, John Axel Eriksson <[hidden email]> wrote:
First a little background:

We've been putting videos in luwak without much trouble for quite a while now, small and a little bigger (by bigger I mean around 6 or 7 megs).

To stream these so far, we've basically done this:

Get the web app to download the video over http from luwak and store on local disk.
Get nginx, through an x-accel, to actually send it to the client (i.e the browser).

For various reasons this isn't really how we want to do it. There is one big reason why we do this currently. We need all clients to authenticate through
our rails-app. So we first tried to have rails send an x-accel to nginx which would then proxy to luwak. This won't work because nginx doesn't support
http 1.1, chunked encoding and all that jazz to backends at least (it's 2011… why I just don't know).

Anyway, for a while now we've thrown node.js into the mix(for other reasons) and recently I started working on some code for node.js to recognize
x-sendfile (similar to x-accel). I also made it do the same type of proxying as nginx can do - but this time it does http 1.1. Unfortunately I couldn't get it
to work. So I started investigating riak/luwak directly. Could we even stream a video out of luwak directly? No we couldn't. So the problem goes all the
way back to riak.

Using Safari it won't stream more than a few seconds and then sound and video stops (though the progress bar keeps moving).
Using Chrome and Firefox I could stream perhaps half the video before more or less the same thing happened.

I then cloned riak and removed one line in deps/luwak/src/luwak_wm_file.erl, specifically line 363 -      {"gzip", fun(X) -> zlib:gzip(X) end}]. So that the list
now only contains [{"identity", fun(X) -> X end}]. (Those are the default_encodings). I will admit I didn't know exactly what I was doing.

Built riak and did some more testing. The results were interesting. Suddenly Chrome and Firefox would play the whole file through (a .webm).
Unfortunately Safari still did the same thing, i.e played 4 seconds and then stopped video and sound (while still showing progress).

I didn't try IE and quite frankly I'm afraid that one might just blow up my computer…

Anyway, I just wanted to ask the list if anyone ever tried the same thing or if any of the good people at basho knows more than I do (which I suspect
they do :-).


Thanks,
John
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



--
Sean Cribbs <[hidden email]>
Developer Advocate
Basho Technologies, Inc.



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com