How does it compare to https://slatedb.io/ ?
Seems similar ideas, although SlateDB seems a bit more lightweight and using Parquet as primitive (even using Arrow) might mean more compute-heavy on client-side?
SPspwa412月24日
This is so weird. If you're using this library
1) your serverless and edge runtime needs to have internet access, so it can contact anyone
2) you're obviously not going to be able to efficiently write to S3 while providing guarantees, so it'll be expensive
3) you're writing in rust, so you really care about correctness and efficiency
This seems like a contradiction. Why would you do this as opposed to hosting a redundant postgres on 2 Hetzner/OVH/... servers and writing to that?
BRbrainless12月24日
Lovely project. Also @rubenvanwyk mentioned SlateDB. I am not sure if this will fit my use-case but, today, I was looking for data hosting options for a self-hosted LLM+bot for email/calendar.
I have this product I have tried and stopped before: https://github.com/pixlie/dwata and I want to restart it. The idea is to create a knowledge graph (use Gliner for NER). Compute would either be on desktop or cloud (instances).
Then store the data on S3 or Cloudflare Workers KV or AWS Dynamo DB and access with cloud functions to hook up to WhatsApp/Telegram bot. I may stick with Dynamo or Cloudflare options eventually though (both have cloud functions support).
I need a persistent storage of key/value data (the graph, maybe embedding) for cloud functions. Completely self-hosted email/calendar bot with LLM, own cloud, own API keys. Super low running cost.
FOfoodbaby12月30日
Super interesting! How does your table format compare to paimon? Would you consider supporting it or iceberg?
WIWilcoKruijer12月17日
Sounds very interesting, but the README has me pondering the downsides. Is the latency very high? Are requests not immediately durable? Is it super expensive?
评论
5 条预览评论 · 正在加载完整讨论请先登录 h4cker 账号，然后连接 Hacker News 后发表评论。
How does it compare to https://slatedb.io/ ? Seems similar ideas, although SlateDB seems a bit more lightweight and using Parquet as primitive (even using Arrow) might mean more compute-heavy on client-side?
This is so weird. If you're using this library 1) your serverless and edge runtime needs to have internet access, so it can contact anyone 2) you're obviously not going to be able to efficiently write to S3 while providing guarantees, so it'll be expensive 3) you're writing in rust, so you really care about correctness and efficiency This seems like a contradiction. Why would you do this as opposed to hosting a redundant postgres on 2 Hetzner/OVH/... servers and writing to that?
Lovely project. Also @rubenvanwyk mentioned SlateDB. I am not sure if this will fit my use-case but, today, I was looking for data hosting options for a self-hosted LLM+bot for email/calendar. I have this product I have tried and stopped before: https://github.com/pixlie/dwata and I want to restart it. The idea is to create a knowledge graph (use Gliner for NER). Compute would either be on desktop or cloud (instances). Then store the data on S3 or Cloudflare Workers KV or AWS Dynamo DB and access with cloud functions to hook up to WhatsApp/Telegram bot. I may stick with Dynamo or Cloudflare options eventually though (both have cloud functions support). I need a persistent storage of key/value data (the graph, maybe embedding) for cloud functions. Completely self-hosted email/calendar bot with LLM, own cloud, own API keys. Super low running cost.
Super interesting! How does your table format compare to paimon? Would you consider supporting it or iceberg?
Sounds very interesting, but the README has me pondering the downsides. Is the latency very high? Are requests not immediately durable? Is it super expensive?