25 messages in ru.sysoev.nginxRe: A hardware question
FromSent OnAttachments
mikeApr 27, 2008 12:36 am 
Dave CheneyApr 27, 2008 1:21 am 
mikeApr 27, 2008 1:58 am 
Dave CheneyApr 27, 2008 2:11 am 
mikeApr 27, 2008 1:34 pm 
Janko HauserApr 27, 2008 1:44 pm 
Igor SysoevApr 27, 2008 9:34 pm 
mikeApr 27, 2008 10:52 pm 
Igor SysoevApr 27, 2008 11:05 pm 
Chavelle VincentApr 28, 2008 3:16 am 
Igor SysoevApr 28, 2008 7:10 am 
mikeApr 28, 2008 11:45 am 
kinglerApr 28, 2008 12:03 pm 
mikeApr 28, 2008 12:26 pm 
Ezra ZygmuntowiczApr 29, 2008 3:11 pm 
mikeApr 29, 2008 4:55 pm 
mikeApr 29, 2008 7:25 pm 
mikeApr 29, 2008 9:11 pm 
Manlio PerilloApr 30, 2008 1:11 am 
mikeApr 30, 2008 2:09 am 
Manlio PerilloMay 1, 2008 2:47 am 
mikeMay 1, 2008 8:17 am 
mikeMay 1, 2008 3:11 pm 
Eden LiMay 1, 2008 6:31 pm 
mikeMay 1, 2008 7:03 pm 
Actions with this message:
Paste this link in email or IM:
Paste this link in email or IM:
Atom feed for this thread
Paste this URL into your reader:
Subject:Re: A hardware questionActions...
From:mike (mike@public.gmane.org)
Date:Apr 28, 2008 12:26:48 pm
List:ru.sysoev.nginx

I think it would be something worthwhile myself.

Just needs parameters - the tracker database info, and the domain right? Then you just use the domain + key to get the file. Maybe domain isn't even required.

However, wouldn't you need some sort of logic to determine how to create the key?

Some people use an md5 of the URI I guess, so it would be something like

foo.com/media/abc72849482bc8398af98effd9g843

/media/ would be mapped to something like

option #1 would be nginx do the mysql connection itself. option #2 would be nginx just connects to the tracker (this is probably best) since it's a simple socket connection

location /media/ { mogilefs on; mogilefs_domain images;

option #1 mogilefs_tracker_username mogilefs; mogilefs_tracker_password foo; mogilefs_tracker_host 192.168.1.203; mogilefs_tracker_database mogilefs;

option #2 mogilefs_tracker_host 192.168.1.202:7001;

and perhaps something like mogilefs_key_style hash; ? or something... (used to determine how the keys are actually setup) }

I think the biggest missing piece of the puzzle is how to determine the key structure. Otherwise I am sure it's quite simple, and it uses dav/HTTP to retrieve the files anyway, so the C code needed would be:

option #1: 1) mysql client 2) something which can do efficient HTTP GETs with offset support (probably already in nginx)

option #2: 1) something that can do a simple socket connect, ask the tracker for info, and then pass it on to step 2 (probably already in nginx) 2) something which can do efficient HTTP GETs with offset support (probably already in nginx)

If I knew anything worthwhile in C I would try it myself. I am sure someone even with basic C skills could hack it up really quick. It just needs one socket to talk to the tracker, then depending on the reply, open a second connection to grab the file...

On 4/28/08, kingler <king@public.gmane.org> wrote:

I know there are MogileFS client libraries for Perl/PHP/Ruby, but if someone can code a C library for MogileFS and it is then possible to make a Nginx module to support. I guess this will be more efficient than going down the Perl/PHP/Ruby route.

Does it make sense at all? Would the benefits outweigh the extra efforts to code the nginx module?

On Mon, Apr 28, 2008 at 11:46 AM, mike
<mike@public.gmane.org> wrote:

I am actually thinking about making a MogileFS passthrough

(I posted an email about it, looks like someone in Japan figured it out)

Would allow for minimal PHP/perl/python/whatever language to be involved, just enough to look up where a file is. Then pass that back to nginx with "file is on $server" and then use X-Accel-Redirect to bounce to that server...

Haven't done it yet, but I think it would work quite amazing!

On 4/28/08, Igor Sysoev <is-G@public.gmane.org> wrote:

On Mon, Apr 28, 2008 at 12:16:59PM +0200, Chavelle Vincent wrote:

Igor Sysoev wrote:

We use the proxying in this case instead of NFS:

client > nginx (1) > nginx

On nginx (1) it's better to set "proxy_max_temp_file_size 0" for the proxied location.

You recommend to use proxying for mass file serving. I am in this case, I use nginx and a specific module that I have coded. (to simplify things, it control access files and redirect client to ngx_http_internal_redirect)

I have think to use NFS for serve the files. I don't understand how to use proxying instead of that.

Set up nginx on NFS server and proxy to it instead of getting files via NFS:

location /files/ { proxy_pass http://nfs_server_nginx; proxy_max_temp_file_size 0; }