I’m interresting in way to find the shortest route on map.

I use MapDotNet and SQL server 2008.

Do you guys have any ebook or article talk about this?

Thanks,
Ho Nguyen

I may actually have a life if I could run my queries, identities, analysis, complex statistic computations.

I’m working on an Ecosystem Based Management plan ( very large areas ) in norther BC and I sure could use another 10, 20… 100 cpu’s running for me.

*sigh*… whoops, an identity just finished, I better run the next one…

Yah, I could easily do with some extra horsepower.

In the process of my research I have developed a conceptual algorithm for the calculation of viewsheds on TIN surfaces. To my knowledge no commercial GIS can compute a viewshed on a TIN or return the results as a TIN, yet the literature suggests line of sight calculations (a core function of viewshed calculations) are more accurate when computed on TIN surfaces. My algorithm utilizes brute force line of sight calculations to determine visibility. When conducted on very large TIN surfaces, this would require large amounts of processing power.

Is this the type of computations you are interested in?

((The classic example is the traveling salesperson problem, where nobody has figured a substantially better way to find the shortest route that visits each city in a list than checking all possible routes between them. The time it takes will grow with the number of possible routes, which is the factorial of the number of cities. Thus if you have 1000 cities, adding a 1001th city will slow the search for the optimal solution by a factor of 1001. It’s clear that constant-multiplier speed improvements, even big ones, are mostly helpless in the face of this sort of performance characteristic.))
There are many heuristic and approximate ways of tackling these when you don’t need the absolute perfect answer, and faster geoprocessing could be very helpful there…

Since you’re interested in redistricting, you might be interested in a similar problem I have some experience in: automated conservation reserve design. You can google up MARXAN for a tool which is widely used in an attempt to identify areas that preserve the most biodiversity while still doing so efficiently. Not very different from trying to capture voting-behavior-similar blocs. (Note: I’m the author of a very similar and now rather out-of-date tool called SPOT.)

I’m certain that similar methods are used to optimize forestry yield/profits, and probably ditto for other agricultural fields. Actually, similar methods are used all over the place by all kinds of industries.

As a practical matter, most of the time this sort of thing is tackled by first using geoprocessing and database crunching to boil things down to a Big Fat Matrix which is then fed into custom-written tools.

When I said that we are looking at different scales of problem I was simply going by your comment above where you said “I grant you that getting into the multiple terabyte range requires much more thought about hardware and overall architecture than working in the hundreds of gigabyte range. But it’s getting very close and it could well be this summer …”. In a later comment you specified a system with 80TB of disk storage, but that is a completely different question from how you can effectively run complex spatial analysis across that much data. I agree with you again when you said that this “requires much more thought about hardware and overall architecture”. So overall I think we’re agreeing about a lot of things .

I smiled at your comment about “the old days when no one could afford a supercomputer so people developed centralized architectures to time-share the few supercomputers that funds allowed to be built”. Have you used Google recently? I don’t think you could afford their hardware, but that doesn’t mean you can’t take advantage of their capabilities.

That’s an analogy that cuts both ways. Sure,I agree that Google is a perfect example of a centralized architecture that time-shares a very expensive resource that otherwise no one could afford to use. But even Google agrees that is not right for everyone and for all applications.

For example even Google understands that time sharing the service it provides is not right all the time, so Google sells their search appliances directly to those users who want to and who can afford to run their own search infrastructure. These are typically large organizations and especially large organizations dealing with sensitive data they would prefer not be accessible, not in any way, to an outsider, not even to a “trusted” outsider.

In general, the more proprietary the data and algorithms the more an organization wants to be sure that no outsider can get their hands either on the data or on the algorithms.

Historically, that’ s been one of the reason for the shift from time sharing of supercomputers to organizations running their own supercomputers. If you are a drug company with billions of dollars invested in your portfolio of molecules, or an investment house invested into computational finance or a political party investing a hundred million into computational electioneering you don’t want to risk anyone else finding out what data you think is worth exploring or how you analyze that data or what your results may be.

Something else to consider is that as a technical matter Google is not very representative of, well, computing. Google does a handful of computationally very simple things on a massive scale. It is not remotely as complex as deep, custom analysis typical of, say, supercomputing or spatial analytics.

So, sure, if you want a very algorithmically lightweight scan of a snapshot of enormously large data, yeah, Google is your thing. If you want to program a heavyweight algorithm like the election studies example given or, say, a protein folding and interaction algorithm making pairwise comparisons between a few million molecules, then Google is a terrible idea.

Those latter cases show other examples of why people often prefer to have supercomputing under their own control instead of time sharing: it is not just privacy, it is the ability to attain much stronger computational capacity.

In a sense, Google is about big disks and small brains. If I understand your target market, you see both big disks and significant brains as well. Historically, people who wanted massive computing intelligence, especially with very fast data access have increased their chances the more that they are able to keep such resources closer to them.

In the present case, I get the feeling that perhaps technology is moving faster than expectations, for instance:

we probably are talking about different scales of problem right now. The solution I am talking about is working with multiple terabytes today and that is where it is especially strong

Well, the $100,000 cluster I advocated earlier was 80 terabytes with 5120 processors so it is true that perhaps I overspecified if you are aiming for smaller tasks, in the multiple terabytes instead of the many tens of terabytes. I apologize if I misunderstood and overshot the requirement.

If all you need do is multiple terabytes, well, then you could get away with a much smaller cluster, say one or two machines for a total of, say, 8 terabytes and 1024 processors. That would keep it well under $20,000, a much more affordable solution.

But if the complaint that what I’m suggesting is so expensive that it must be time shared, well, I’d offer two observations:

First, the sorts of customers you mention have plenty of money and can easily afford $20,000 or $100,000. They won’t hesitate to spend that even if a time-shared solution is less expensive, because no way, no how will they give up the power of having 1024 to 5120 and more processors all for themselves.

Second, although I grant you that spending $100,000 to get 5120 processors and 80 terabyte capacity sounds like a lot, that’s only at today’s prices. A year from now it will be $60,000 and the year after that $30,000.

Large customers tend to have slow procurements so by the time a lot of these folks get going it will probably be down to $25,000 with storage and computing capacities even higher than today. Heck, in two or three years we could be talking 300 terabytes and 15,000 processors for that money!

So no, I don’t knock the idea of time-sharing an expensive resource to make it available to more people. But I am saying that both security and performance are strong reasons for not time sharing and that the costs of direct, distributed technology are getting so low that it is already probably too inexpensive (as has become the case with supercomputing) to bother time sharing anyway.

Could you comment whether your plans are for specialized hardware, or is it software not tied to a specific hardware platform?

Actually, the US Supreme court says congressional redistricting can occur whenever the state assembly wants it to (here).

As far as “unknowns” —I would kindly offer that you vastly underestimate retail market data. Do you own a GM with OnStar or use MasterCard / Visa? Supermarket club card? There are no unknowns – only the un-calculated (apologies to Webster’s and my 7th grade English teacher).

Bad deadline, however. Redistricting won’t take place until 2012.

Objective: optimized area allocation into ‘n’ voter districts for either competitive or non-competitive election cycles (depending on your view of the democratic process).
Secondary by-product of above: model trend impacts based on media buy cost to optimize campaign spending and general predictive spatial consumer behavior based on “injected” catalysts.

Input 1: partisan voting history by precinct areas (discrete partisan votes totals with precinct x precinct ratios for turnout, total registered voters, total eligible to register, and total population)

Input 2: population migration (historical, current estimated, and predicted) by census blockgroup or even census block if you have the computational horses.

Input 3: aggregated consumer behavior records by multiple dimensions – usually aggregated by zipcode or block group (not a fan of zipcode area analysis – but that’s usually where the biz data is scaled to)

Input 4: discrete and aggregated campaign contributions from the FEC database

Input 5: other spatial market segmentation data to control for on-demand measures (crime, education, vehicle type, etc.).

Input 6: political subdivision boundary data to constrain allocation model behavior and enforce current county and municipal communities of interest.

The exponential nature of allocation modeling at the state level for sub-county areas has generally been categorized as “computationally “NP hard”" …. ergo the value of Mr. Batty’s “machine”.

If this working “on-demand” model were available it would be a gold-mine for political interest groups and consumer marketing firms.

Of course there are models out there, but they frequently abstract or overly generalize the spatial dimension — this model would be spatially accurate to the neighborhood level at each given point in time loaded into the model. And the real value is in the predictive ability by location based on a combination of “injects” into the consumer environment (incl. gas prices, severe weather, approval ratings, etc.).

I imagine it like a giant climate or weather model only the gradients represent a given spatial behavior across time instead of temperature, precipitation, etc.

If you could put a simple GUI on the front end and have it ready by July or August (in time to have fun before the Nov. election) – that would be great!