for my part, I can read and understand in part code in C++ if I have a box of paracetamol near me hahaha
I've heard a lot of good things about Go and it's crossed my path a few times, but I never got into it.
I wrote the image server in Go
Starting from the git repo of Steem Imagehoster that you have adapted or starting from zero?
What are the hardware specs of the Blurt image server? I would also be curious to know how much space the storage of the images takes?
It would be nice if one day we could have a decentralized image storage solution to beat STEEM and HIVE in this field, too many people still consider the images of the posts as without added value for the blockchain because not decentralized.
And once again I thank you for the work provided which contributes to give more value to the Blurt blockchain 👍
Just to add to my earlier comment: the main reason I switched back to the original Blurt image servers with the separate uploader/proxy/server is because the Steem/Hive Imagehoster stores local copies when it proxies remote images. The pro of doing that is that, if the remote image gets deleted, it will still show up on the blog since there is a local copy of it. The con of doing that is the enormous storage space used and the cost of storage.
I opted for the space/cost savings. Our proxy only proxies images; it doesn't save a local copy.
Yes Hive had to change its server because of this (168TB RAID drive now), but the advantage as a user is huge. When I first got on Steem I started by using the Steepshot Steem dApp (which stored the images itself) which has since closed, I lost all the images from my posts from that time hahaha
Ooh i had a video chat with the Steepshot team once
As of a few seconds ago, the images add up to about 520 GB of space.
Thanks, far below what I expected :) Another one, I forgot to ask you hahaha
What is the maximum accepted weight you put for the images? Or maybe you do a resizing on the server side?
That'd be interesting to know.
Is IPFS used on the image server as well?
Personally I would prefer Arweave for immutable or Filecoin if we need GDPR compliance to take down images.
Cc: @tekraze, @saboin
We use S3-compatible storage bucket, the same as what they use for Steem and Hive.
We had plans to switch over to uploading on IPFS as some point, but we never went through with it.
As @tekraze said in his comment, IPFS is slow and unreliable. We would have to run IPFS nodes as well, which would cost a lot more than using an S3-compatible storage bucket.
No, it's a storage bucket
A lot of blockchain tech uses IPFS, is there any reason for not doing so with Blurt?
IPFS is slow as well not reliable, it was used at start but then was removed.
Don't 3speak/dtube use IPFS for storing their videos? 3speak seems pretty fast and has been reliable on everything but upload.
Dtube storage is not working yet(under maintenance and some work is going on for IPFS) and 3speak use self-hosted IPFS that is a little performant but costly over what current solution we have.
The Steem/Hive Imagehoster is an all-in-one server/uploader/proxy. On Blurt we use three separate programs for those functions. Blurt did use the Steem/Hive Imagehoster for a while, but when I came on board, I rewrote the uploader, tweaked the proxy, and built a server from scratch in Go that just fetches the images from our S3-compatible storage bucket and serves it to the client.
So to answer the question simply, I wrote the image server from scratch in Go in an afternoon and launched it so that there would be something. At the time, the Blurt infrastructure was in shambles because a lot of it was run by a dev who had left Blurt and turned off his servers. This meant that any image that was uploaded through the blurt.blog UI didn't show up on the blogs, image upload didn't work, there was a lot of stuff that didn't work. I should make a post someday to rehash this history and tell the story of how and why I got recruited to the core team.
It runs in a Docker container that is setup to restart if it crashes, so I wasn't too concerned about error handling since it would just restart automatically if something went wrong. And to my surprise, it didn't crash even once.
The proxy, on the other hand, is written in JavaScript, has error handling and crashes every few days. We don't notice it because it also runs in a Docker container that is setup to auto-restart when it crashes.
The image uploader code is here (I'm going to update the README.md file over the weekend).
The image proxy code is here (this one also needs an updated README.md).
The image server code is here (I should copy that to the Blurt repo and update README.md on this one, too).
Over the weekend, I plan on updating the REAMDE.md files for all of the components of the Blurt infrastructure with instructions on how to deploy. I plan on making some posts that will layout the infrastructure that we run as the Blurt core team.
Thanks a lot for this very interesting feedback 👍
For my part in one project for a customer I use imgproxy in a Docker container, and I am quite satisfied with it.
You seem to know docker really well! For my part, I have always had trouble getting into it and am far from being a specialist hahaha
I would be interested to read it ;)
Thanks for the links :)
That image proxy looks very interesting. I've been wanting to rewrite some of our infrastructure in Go for the performance gains. Since all our infra is proxied through Cloudflare CDN, I'm not sure there would be that much difference by having more performant image proxy and server, but I'd like to test the JS version against the Go version to see how much difference it makes.
I'll bookmark that image proxy and look at it later in case I ever get around to rewriting the image proxy in Go.
I think @jacobgadikian was using that specific proxy before I came on the Blurt core team. It saves copies of remote images in the S3 bucket, so that would take up a lot of space.
When I copied the images from Jacob, a lot of the space taken was from the proxy saving remote images. I prefer just a plain proxy that doesn't save anything for the reasons I mentioned in my other comments.
I think Go is a good way to explore for performance gains 👍