Static Download of the World

[DocFlareon]

I am beyond annoyed by the words "Receiving Land (Big Number)" whenever I teleport to a location that I haven't visited recently. I would love to have the option to dowload all of the game's assets to a permanent location, rather than streaming into a temporary cache.

[Racktor]

I'd love this to be implemented... I headdesk whenever my cache desides to reset on me, with all the loooooong land loads. x.x

[jerk-o]

Downloading all the terrain ahead of time (or at least all the terrain for and around the starting islands, racial towns, and settlements) would be very nice.

[AmonGwareth]

This is actually something we are working on and I think might be part of the Alpha client that we recently announced for testing.

The reason this wasn't originally done was because Istaria can do dynamic terrain and world changes. Some vets might remember the fun during the Server Merge in 2004 when towns were destroyed, volcanos appeared, and much more. That is the power of the client. But, the trade-off is that terrain and world objects must be streamed.

[Vahrokh]

This is actually something we are working on and I think might be part of the Alpha client that we recently announced for testing.

The reason this wasn't originally done was because Istaria can do dynamic terrain and world changes. Some vets might remember the fun during the Server Merge in 2004 when towns were destroyed, volcanos appeared, and much more. That is the power of the client. But, the trade-off is that terrain and world objects must be streamed.

Considering how often dynamic terrain is being used, it's being more of an annoyance than a feature. Plus imho back in time there had been a design mistake. We got a game CD (therefore plenty of room to store files) and dynamic terrain would always be a < 1% change. Therefore the mapping technology should have focused at having us a "default terrain" not a "no terrain" cache that would eventually be updated to the next "stable state" at the next patch involving terrain changes (i.e. rarely) and only updated once. The dynamic terrain would have been an override of the stable state, either temporary or permanent (= new stable state).

That way we'd have both dynamic terrain and no need to download it again and again except for just a small subset to download once and only every terrain change patch.

Edit: that would involve terrain blocks versioning but considering the vast saving in bandwidth it'd more than justify the slight programming effort to do that.

[AmonGwareth]

Considering how often dynamic terrain is being used, it's being more of an annoyance than a feature.

Exactly, that is why we are changing it.

[Guaran]

Considering how often dynamic terrain is being used, it's being more of an annoyance than a feature. Plus imho back in time there had been a design mistake. We got a game CD (therefore plenty of room to store files) and dynamic terrain would always be a < 1% change. Therefore the mapping technology should have focused at having us a "default terrain" not a "no terrain" cache that would eventually be updated to the next "stable state" at the next patch involving terrain changes (i.e. rarely) and only updated once. The dynamic terrain would have been an override of the stable state, either temporary or permanent (= new stable state).

That way we'd have both dynamic terrain and no need to download it again and again except for just a small subset to download once and only every terrain change patch.

Edit: that would involve terrain blocks versioning but considering the vast saving in bandwidth it'd more than justify the slight programming effort to do that.

Good points. The client guys are working on it. The only thing which would have to sit outside this is the data for the player plots. The plot structure data is really the only thing that needs to remain using the streaming method. Those are constantly changing, and it isn't likely efficient to try and preload all that. But preloading the land itself yes. Part of the client rework is to package the terrain into zip files, to reduce the number of files. As it is a fully loaded world cache is > 100,000 files, maybe approaching 200,000. It's too much for windows to handle quickly and contributes to lag and load times.

I will be glad to test the new client when it's ready.

[Vahrokh]

Good points. The client guys are working on it. The only thing which would have to sit outside this is the data for the player plots. The plot structure data is really the only thing that needs to remain using the streaming method. Those are constantly changing, and it isn't likely efficient to try and preload all that. But preloading the land itself yes. Part of the client rework is to package the terrain into zip files, to reduce the number of files. As it is a fully loaded world cache is > 100,000 files, maybe approaching 200,000. It's too much for windows to handle quickly and contributes to lag and load times.

I will be glad to test the new client when it's ready.

Even the downloading of structures may be optimized by attaching a simple versioning system on their "blocks". This would even make it possible for "affordable" interior forniture to make taverns / guild houses less "desert inside". The forniture would require (the painful part) a tweak in the occlusion code to simply hide it when looking from the outside - to keep FPS viable. "If IsForniture = 1 and player vector outside building bounding box then don't render"

By attaching a simple version number to each block, only the changed ones would require a download, this means few blocks per days even on trafficked areas.

[Akrion]

Even the downloading of structures may be optimized by attaching a simple versioning system on their "blocks".

Simple in theory, anyway. Either every individual structure would need their own version, or their own ID. This version/id would need to be stored on your computer along with the location and orientation of every individual structure. Not only that, but every time you enter an area with plots/lairs the client would need to figure out which structures in the cache are located near the player (Which means it would have to scan every structure in the cache lest it check every structure with the server each time you port) and check each version of said structures with the server to make sure they're all up to date, otherwise there would be no way for the client to know when and what it needs to update in the cache.

In the end, I don't think that would be more optimized than the way it's already implemented. Since player structures can always be changing, re-downloading information regarding them isn't so bad. It's the re-downloading of stuff that rarely changes that can be unnecessary, which is great to hear that it's being addressed. (Thank you for that one, Devs.)

[Vahrokh]

Simple in theory, anyway. Either every individual structure would need their own version, or their own ID. This version/id would need to be stored on your computer along with the location and orientation of every individual structure. Not only that, but every time you enter an area with plots/lairs the client would need to figure out which structures in the cache are located near the player (Which means it would have to scan every structure in the cache lest it check every structure with the server each time you port) and check each version of said structures with the server to make sure they're all up to date, otherwise there would be no way for the client to know when and what it needs to update in the cache.

In the end, I don't think that would be more optimized than the way it's already implemented. Since player structures can always be changing, re-downloading information regarding them isn't so bad. It's the re-downloading of stuff that rarely changes that can be unnecessary, which is great to hear that it's being addressed. (Thank you for that one, Devs.)

Every structure already have their own instance id, every structure has already to be searched for and fetched when in sight distance.

Now, the whole system is not totally trivial to make but I'd be able to write it, so I suppose their dedicated developers can do that as well.
Of course all of this is pure theory mode, as I don't believe they have resources available for this (even if it's years that players ask for forniture, it's why I talk about it at all).

[Racktor]

There any update to this at all?

[Dhalin]

This sounds like one of the best things that could potentially happen to this game I've heard for... awhile.

I'd much LOVE to see something like this happen.

[Steelclaw]

The bits and pieces for a static world cache are coming in to place. The world cache covers the terrain and LOD tiles for the terrain. World objects will need to continue to be streamed.

The two major (but not the only) issues are ascertaining the stability of the static world cache and package code in the alpha client; and figuring out a deployment mechanism for the static cache.

The latter is a topic for discussion, since the deployment would likely result in a 0.7 - 2.5GB client patch once the static world cache is deployed, and of course, increase the initial download footprint by that much as well.

I guess now is a good a time as any to open a discussion to explore the potential impact of pushing out the extra data upon the entire player base, especially with respect to download caps.

Let's start with something simple within these bounds: assuming the implementation and deployment of the static world cache went as described above (0.7 - 2.5GB patch, bigger initial download footprint), who will be negatively affected (compared to current streaming terrain downloads), and how?

Please focus on the impact of the download size, not on alternatives to make the download smaller. No hypothetical people either -- we've gone over plenty of hypothetical situations internally and need some real sample data instead. (Really, how many players out there play over GPRS/Edge/3G with a 2GB/month bandwidth cap? )

[Dhalin]

No issues with a 2-3GB patch here, but isn't there a way to make it so you "download on demand"?

Instead of downloading the whole world at once in a huge 2.5GB patch, couldn't we do what we do now, download-as-we-go, and only the areas we go to? Except this time, the static content we download, gets kept instead of thrown out.

[walkerglassmire]

A minor related query, will this mean that the bug that corrupts the world cashe for XP users be squashed then? I know for me at this point... I can't port anymore if I've been somewhere before.

[Ettanin]

Additional idea:

Leave the streaming system as is but:
1st: Do not limit the world cache size
2nd: Allow logged in players to download the world data every 24 hours directly from the shard via HTTP. This contains server sided regional changes.
3rd: Override the local cache if there's a regional change server sided and update the cache as recent.

Advantage:
- People with highspeed internet access will load faster as they can fetch the entire world data.
- People who have a slow internet connection will load areas foreign to them only on demand.

[meepsa]

No issues with a 2-3GB patch here, but isn't there a way to make it so you "download on demand"?

Instead of downloading the whole world at once in a huge 2.5GB patch, couldn't we do what we do now, download-as-we-go, and only the areas we go to? Except this time, the static content we download, gets kept instead of thrown out.

I vote for this, if it's possible.

though I have no problem with downloading it all at once, I can understand that people with slower internet cons would.

[Spirit]

I have a monthly 10gb download limit... however between midnight and 8am that limit doesn't apply, so I generally do big patches/downloads overnight. No big deal here at all, as long as I remember to get the patch going in advance!

Rakku

[Racktor]

No issues with a 2-3GB patch here, but isn't there a way to make it so you "download on demand"?

Instead of downloading the whole world at once in a huge 2.5GB patch, couldn't we do what we do now, download-as-we-go, and only the areas we go to? Except this time, the static content we download, gets kept instead of thrown out.

That's a good idea really. Would throw out the patch-size issue~

[Reflection Rip]

Additional idea:

Leave the streaming system as is but:
1st: Do not limit the world cache size
2nd: Allow logged in players to download the world data every 24 hours directly from the shard via HTTP. This contains server sided regional changes.
3rd: Override the local cache if there's a regional change server sided and update the cache as recent.

Advantage:
- People with highspeed internet access will load faster as they can fetch the entire world data.
- People who have a slow internet connection will load areas foreign to them only on demand.

I agree. Still let the players stream the data as it is now, with a possibility of updating the client/cache without logging in. And do a revision control system on all the areas(chunks), and player lots.

This way an active player will only need to download any recent changes. This should actually decrease the lag, load times, and server demand. A win not only for the players but for the devs too.

An RCS or CVS system should be possible for player lots. The system does not need to keep the current state of each structure, just if the state has changed. If the state has changed then update the client for every structure (changed or not).