Value streams

[ivakegg]

We are starting to talk about schemas where we would potentially have huge values associated with relatively small keys. In this case it would great to be able to get keys and a stream for the value back instead of pulling the full value into memory. Perhaps this could be configurable with an optional threshold over which a stream would be returned instead of the value bytes. I could see the Value object being modified with a "isStream()" and "openStream()" methods.

[dlmarion]

Is the scope of this request on the server-side only? I don't see how we pass a stream of bytes from the server to the client.

The full value, compressed via rfile compression, is already going to be in memory because it's in the block cache. I think there would be a lot of code that would have to change in RFile to not decompress the Value, if that's even possible because the entire block is probably decompressed to extract the Keys and Values contained within it.

Have you considered compressing the object before writing it into the Value?

[ctubbsii]

To add to what @dlmarion said about the challenges of avoiding reading the values from RFiles, if all you want is to filter out the values when sending back through the iterator stack and then back to the client, that's pretty trivial to do with an iterator. We have one that does this, called SortedKeyIterator, that just replaces the value with an empty one.

[ivakegg]

I guess our main use case we are thinking of is on the server side itself. I see how creating a stream on the client side would require a bit more plumbing but that is certainly doable. We are considering cases where even compression may not be good enough.

[ivakegg]

For the client side, you would pass info to the client such that the client could open the stream (e.g. the rfile location, and offsets to the value.)

[dlmarion]

It might be easier to store the compressed value in a different file then if you are just going to return the location to the client and then do another read from the client. Using a different file will at least give you a static location as the location returned from the server to the client as part of the scan will change due to compaction, and may change during the scan due to a concurrent compaction.

[ivakegg]

After several discussions it appears that this is not a feature that Accumulo would take out for several reasons:

1. The underlying infrastructure of the BCFile and RFile does not lend it self to this time or processing.
2. The data is going to be pulled and cached in memory anyway in the form of HDFS blocks.
3. There are other mechanisms by which we can handle storing large blobs in separate files to solve this type of problem.

[keith-turner]

It might be easier to store the compressed value in a different file then if you are just going to return the location to the client and then do another read from the client.

Could create a prototype to do this w/o making any changes to Accumulo internals. There are two things that would be missing doing it this way that you get when storing the data w/ the key in Accumulo. First would not have automatic GC of these files when they are no longer referenced. Second the file would not have any visibility constraints. Possible workaround for visibility would be to encrypt the file and store the encryption_key+url in the Accumulo value. Then can only see the encryption key when you have the correct authorizations. For GC, could create a summarizer in Accumulo that counts URLs in use. Then could use these summaries to find files with no references. However this may only work well if the number of URLs is somewhat small relative to the overall table size.