Luwak PUT Content-Range

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Luwak PUT Content-Range

John Axel Eriksson
Is it possible to incrementally add to a file in Luwak using PUT and the Content-Range header. I just assumed that it was but I can't seem to
get the expected results, it just overwrites whatever the key contents were before. The reason I want to do this is because we have some pretty
large files I don't want to load fully into memory before PUTing them.

John

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak PUT Content-Range

bryan-basho
Administrator
On Tue, Nov 29, 2011 at 6:07 AM, John Axel Eriksson <[hidden email]> wrote:
> Is it possible to incrementally add to a file in Luwak using PUT and the
> Content-Range header.

Hi, John.  At this time, it is not possible to use Content-Range when
PUTting Luwak content.

> The reason I want to do this is because we have some pretty
> large files I don't want to load fully into memory before PUTing them.

I can *almost* help you with this case, though.  POSTing to a Luwak
file is intended to function as "append".  Unfortunately, a bug
prevents this from working properly.  Once this bug is fixed, you
should be good to use POST for this case:

https://issues.basho.com/show_bug.cgi?id=1295

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak PUT Content-Range

Greg Stein
In reply to this post by John Axel Eriksson


On Nov 29, 2011 5:08 AM, "John Axel Eriksson" <[hidden email]> wrote:
>
> Is it possible to incrementally add to a file in Luwak using PUT and the Content-Range header. I just assumed that it was but I can't seem to
> get the expected results, it just overwrites whatever the key contents were before. The reason I want to do this is because we have some pretty
> large files I don't want to load fully into memory before PUTing them.

Hmm? You can read from a file and write to the http socket. There is no reason or need to load the entire contents into memory.

I don't know what client you're using, but I do know the Python client is broken in this regard. It erroneously loads the full content into memory. But there is nothing from the Riak server that demands such an approach.

Cheers,
-g


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak PUT Content-Range

John Axel Eriksson
Well, I don't know what I'm doing wrong here but I'm mainly using ruby. Excon (which Ripple uses) doesn't seem to have a way to pass an IO to a put request (am I missing something?).

I've also tried NetHTTP in ruby which is actually supposed to be able to do that and I've tried Javas HttpURLConnection in JRuby (I'm really not a Java expert). None of them works unfortunately:

# I tried this

require 'net/http'

## this doesn't work and doesn't store all the data
path= "/path/to/file/here" ## 88 MB file
key = "large_file"
req = Net::HTTP::Put.new("/luwak/#{key}")
req.body_stream = File.open(path)
req.content_type = "video/mp4"
req.add_field 'Transfer-Encoding', 'chunked'
res = Net::HTTP.start(uri.host, uri.port) do |http|
  http.request(req)
end

# And this - in this case riak started thrashing like crazy until
# I just killed it (it never stored more than  1 MB of data though)
path= "/path/to/file/here" ## 88 MB file
key = "large_file"
req = Net::HTTP::Put.new("/luwak/#{key}")
req.body_stream = File.open(path)
req.content_type = "video/mp4"
req.content_length = File.size(path)
req.add_field 'Transfer-Encoding', 'chunked'
res = Net::HTTP.start(uri.host, uri.port) do |http|
  http.request(req)
end

# And in JRuby I tried Javas HttpURLConnection

require 'java'

include_class "java.net.URL"
include_class "java.io.RandomAccessFile"
include_class "java.io.FileInputStream"
include_class "java.net.HttpURLConnection"

CHUNK_LENGTH=1048576

c = u.open_connection
c.request_method = 'PUT'
c.fixed_length_streaming_mode=File.size(FILE)
# c.chunked_streaming_mode = CHUNK_LENGTH  ## also tried this
c.do_input = true
c.do_output = true
output_stream = c.output_stream
error_stream = c.error_stream
random_access_file = RandomAccessFile.new(FILE, 'r')
bytes_read = 0
buffer = Java::byte[CHUNK_LENGTH].new
begin
  while(bytes_read >= 0) do
    bytes_read = random_access_file.read(buffer)
    puts "bytes_read: #{bytes_read}"
    if bytes_read>=0
      ## the second write here gets an Exception - I think it's a 400 from riak
      puts "writing..."
      output_stream.write(buffer,0,bytes_read)
      sleep 1
    end
  end
rescue Exception => e
  puts "Exception: #{e.message}"
end
puts c.response_code rescue nil
puts c.response_message rescue nil
random_access_file.close
output_stream.close
c.disconnect



On Tue, Nov 29, 2011 at 10:53 PM, Greg Stein <[hidden email]> wrote:


On Nov 29, 2011 5:08 AM, "John Axel Eriksson" <[hidden email]> wrote:
>
> Is it possible to incrementally add to a file in Luwak using PUT and the Content-Range header. I just assumed that it was but I can't seem to
> get the expected results, it just overwrites whatever the key contents were before. The reason I want to do this is because we have some pretty
> large files I don't want to load fully into memory before PUTing them.

Hmm? You can read from a file and write to the http socket. There is no reason or need to load the entire contents into memory.

I don't know what client you're using, but I do know the Python client is broken in this regard. It erroneously loads the full content into memory. But there is nothing from the Riak server that demands such an approach.

Cheers,
-g



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com