Comments on: ArcGIS Map Server Cache in Amazon S3

By: Adam

Adam — Fri, 06 Feb 2009 13:40:40 +0000

Brett,
We ended up writing a windows program that takes a series of parameters (folder on the server, location on Amazon, etc.) and then converting the disk based folder structure to the rest endpoint folder structure, and load it all into s3 at once. The only difference in the column, row folder/file names is: The rest folders are decimal values, and the disk folders are in hex.

By: Brett

Brett — Tue, 30 Dec 2008 20:48:11 +0000

To focus my previous question… anyone know how to set up the rest endpoints on s3?
I realized that once this is done, you just have to set the tileserver options when you add your tiled layer and you are all set.

By: Brett

Brett — Mon, 29 Dec 2008 21:05:18 +0000

Anyone with advice on how to integrate tiles hosted on s3 into an application built on the esri javascript api and dojo?
I’m certain that some of our departments would be interested in this idea, but we are just barely digging into the javascript api at this point.

By: Lakshmanan

Lakshmanan — Sat, 20 Dec 2008 05:47:49 +0000

I’m great to see this post. I have never got chance to store caches in S3 or any other external servers My question is how much safe (i.e legally) for our data. Sorry for this naive question but still like to know more about this.

By: josh

josh — Fri, 19 Dec 2008 23:27:31 +0000

Just as a note, you can get at least two domains with s3 proper, making for 4 synchronous requests from IE7 and FF2 (and 12 from most modern browsers — see http://stevesouders.com/ua/)

To get both urls going with default s3, just follow the instructions at http://docs.amazonwebservices.com/AmazonS3/latest/index.html?VirtualHosting.html for linking up your cname to your bucket, and then use both it and the s3.amazonaws.com domain.

This assumes, of course, that you don’t mind your clients potentially noticing one of your tile urls having s3 in it for the extra push.

By: Jason

Jason — Fri, 19 Dec 2008 22:56:55 +0000

We have been doing this for quite some time now for our clients and it has been working out great. Check out my blog post where I outlined a interesting case study. We use bucket explorer to upload our caches and it allows you to set permissions on the entire bucket and then everything put in that bucket inherits those permissions.

http://www.roktech.net/devblog/index.cfm/2008/8/14/Map-Cache-Tile-Hosting

I’m not sure how many pipes the js api will handle, but we have used 4 ’tileservers’ with no problem. So, we use our servers in conjunction with S3, make for very fast and reliable tile loads.

By: Brian Flood

Brian Flood — Thu, 18 Dec 2008 19:35:37 +0000

@alper – yea, they are rolling out the GAE billing now (beta) so quota limits won’t be an issue going forward. but a clever idea anyway

A2E Cloud Services are coming along very well, I have about 5 blog posts piled up that I will get out next month.

@bruce – tilecache on EC2 is great. fwiw, although its not built in to ArcMap, Arc2Earth V2.1 Map Tile Layer allows you create your own tile layer now (before it was hard coded to VE, Yahoo etc). So you can add any url template and it will download/cache/display the tiles as long as they conform to the commonly used world mercator spec. (anything you can display in GM/VE)

By: Bruce Rindahl

Bruce Rindahl — Thu, 18 Dec 2008 17:11:52 +0000

I am migrating my SVG mapping applications to an EC2 machine running TileCache. The tiles are stored on various S3 buckets and so far the performance is better than our server. What would be nice is for ArcMAP to connect to something like WMS-c or TMS directly.

By: Alper Dincer

Alper Dincer — Thu, 18 Dec 2008 14:49:15 +0000

Hey Brian,

Thanks for tip. There is also another reason to use multiple application, which is not to pass the CPU and other limits

By the way, we are looking forward to see the new version of Arc2Earth, “Cloud Services”. Is there any released date for beta version?

Thx.

By: Brian Flood

Brian Flood — Thu, 18 Dec 2008 12:14:34 +0000

hey all

@alper – you can assign multiple cname entries for a single GAE application. you do have to register and associate a domain first but then you can add multiple sub-domains that all point at your main GAE app (say tile1.yourdomain.com, tile2.yourdomain.com etc). this allows you to have a single cache with multiple subdomain entry points to get around the browser limits
http://code.google.com/appengine/articles/domains.html
http://www.google.com/support/a/bin/answer.py?answer=91080&hl=en

S3 is great for tile caches (Arc2Earth will automatically publish to an S3 account) but I’ll agree with Matt and Jason on the single sub-domain access. I asked them about this when S3 was first available, I think there was a good security/technical reason why they did not offer multiple subdomain and did not plan to in the future. (wish I could remember it right now

CDN – full auto update of the cache would be great, in the mean time you could setup a simple EC2 app to publish against. It writes to S3 and then manually informs the CDN about updates (not just newly added tiles)

cheers
brian