Plugin to store files in Amazon S3 rather than database as blob.

Question

I need a Plugin to store files in Amazon S3 rather than the database as a blob.
Can anyone build this plugin? I can pay for it.

The plugin should work on any version of Q2A.

Comments

What is the advantage of this?

---

this will boost the performance of mysql server as image stored as blob in database creates more I/O operations which leads to decrease in performance.

If the images will be stored in S3 then we will have two advantage as I noticed:
1. Optimize mysql server performance as I/O operations will be less.
2. Using S3 to store image will make our site fast as we can use CDN with S3 which will boost load time of site.

Correct me if I'm wrong

---

But there is an option to store images on disk right? Also, CDN like cloudflare can be used to reduce the traffic to the server. I'm not sure how much use S3 can give here -- if there is a surety this can be implemented.

---

I think your points are correct! Would love to see a plugin myself!

Storing on disk is of no value if any Auto Scaling or Load Balancing is used.

Until there is a plugin what you can do is to cache files like imgs via Elasticache, therefore the db is less stressed.

---

yes, like on my site more than 70% of the image data is served from the cache and the blob access hardly contributes to server load.

Answer 1

This isn’t a direct answer to your question but storing images on S3 will not give you the performance improvement you’re expecting. In fact it will make it slower.

The reason is that all image URLs go via the database, they never link directly to files even when you store files on disk. In other words:

• You have a URL like example.com/?qa=image...
• That runs a PHP script same as any other page on Q2A.
• It checks the database for the referred image.
• The script echo’s the binary data stored, whether in the db or by reading the file from disk.

If you store the image on S3 then your server has to go and read the image and output it to the user. That’s obviously slower than the server reading straight from its own disk.

However, it is possible to use a CDN. I don’t think S3 is really intended as a CDN, CloudFront is the CDN part. That can be put in front of any website, you don’t need to put images elsewhere. It’s quite expensive though so you might want to try something like CloudFlare which is free for basic usage. There are others like MaxCDN, Akamai, CDN77.

Comments

Thanks @scott for answering this question. However, Cloudflare also do not cache image with query string as in q2a
You have a URL like example.com/?qa=image...
Is there any way to cache image for q2a format.

---

Hmm, good point. That's one of the things I've considered changing - i.e. making the URLs static looking like example.com/image/1234567890

---

Is it possible for you to change this in 1.8.4 now? Because it is most important for everyone.
Or as a plugin

---

Cloudfront has support for query-strings, the problem I found is that content in Q2A is "hidden" very often in the HTML output and displayed with JS in the DOM. So, it's not easy to change the URL to use a CDN.

I use the sub_filter directive in Nginx and I'm successfully using Cloudfront in Q2A for JS and CSS files (qa-plugin, qa-theme and qa-content), but I can not use it for images.

---

Can you give some examples where content is hidden? I think most JS updates go through the normal functions and the theme - for example the “show all comments” does an AJAX request which returns HTML.

---

@Scott I always thought the images were cached even with query string. Below is the help from Cloudflare. Am I missing something?

What are Cloudflare’s Caching Levels?
You can set Cloudflare’s CDN to cache static content according to these levels:

No Query String: Only delivers files from cache when there is no query string.
Ignore Query String: Delivers the same resource to everyone independent of the query string.
Standard: Delivers a different resource each time the query string changes.

---

Got it. It is not the query string, rather it is due to images being served as a dynamic resource.
https://support.cloudflare.com/hc/en-us/articles/202775670

---

The "show all comments" HTML is "hidden" in some way. To be more precise:

- Visit a Q2A page and press CTRL-U, you will see the HTML code.

- Expand to see "all comments", if you press CTRL-U you will see the same HTML code, here the HTML of that portion of comments is hidden. AJAX only updates the DOM and that's ok, that's expected. The problem is that sub_filter in Nginx seems to have no effect over AJAX requests/responses.

So, how can I change the URL in this "dynamic" content?