Static Download of the World

[Fyrethorne]

Alpha client? Is there a seperate thread for this? What will it mean?

[justmissu]

The reason this wasn't originally done was because Istaria can do dynamic terrain and world changes. Some vets might remember the fun during the Server Merge in 2004 when towns were destroyed, volcanos appeared, and much more. That is the power of the client. But, the trade-off is that terrain and world objects must be streamed.

[Starstilanxs]

Additional idea:

Leave the streaming system as is but:
1st: Do not limit the world cache size
2nd: Allow logged in players to download the world data every 24 hours directly from the shard via HTTP. This contains server sided regional changes.
3rd: Override the local cache if there's a regional change server sided and update the cache as recent.

Advantage:
- People with highspeed internet access will load faster as they can fetch the entire world data.
- People who have a slow internet connection will load areas foreign to them only on demand.

I don't agree with this, personally, Ettanin. Here's why:

This this one of the major causes of lag in the game, constant streaming of assets which, in many cases in which I have experienced, decides to halt it's progress and sit there with a terrain download queue until you relog. When you do relog, it seems to like to delete every bit of cache that you originally obtained and you have to reload into -every- area again.

Dev's not using the dynamic world that the game is was made to run on makes streaming useless, in my opinion. Once a patch goes live, we just have to download a few more things and that's it.
* No lag
* No streaming of world content as it's already downloaded with the patch update
* No frustration and worry about the fact that you can't do anything when your character gets pummeled while you sit idly by waiting for streaming content
* Lower bandwidth costs for the company since less data is -constantly- sent to the clients.


All of this considered, this is only -my- opinion.

[Ettanin]

Dev's not using the dynamic world that the game is was made to run on makes streaming useless, in my opinion. Once a patch goes live, we just have to download a few more things and that's it.
* No lag

Wrong:
Download only: Server load burst. Lag for everyone;
Stream only: Clientside lag only for the player.
Streaming on demand is better.

* No streaming of world content as it's already downloaded with the patch update

So exclude low-bandwidth and bandwidth limited players as well as UMTSers? Bad idea imho. Streaming is fine as it lowers the bandwidth demand on all players. Why do you need the aerial data of Dralnok's doom for example if your character is only L10? Why needing Aradoth right now if you are at NT for the first few levels?
Really, not everyone has 1337Mbit/s. That's why there should be both options: Streaming AND downloading, at the player's choice, where downloading is available only once per 24 hours per player to prevent bandwidth hogging + lags.

* No frustration and worry about the fact that you can't do anything when your character gets pummeled while you sit idly by waiting for streaming content

See above.

Lower bandwidth costs for the company since less data is -constantly- sent to the clients.

Wrong. Whether you burst load bits or stream them constantly, the amount is the same in the end.
Streaming is even better regarding those 14 days trialers who wont come back after these 14 days, as you only load the client partial instead of fully.

[Starstilanxs]

Wrong:
Download only: Server load burst. Lag for everyone;
Stream only: Clientside lag only for the player.
Streaming on demand is better.


So exclude low-bandwidth and bandwidth limited players as well as UMTSers? Bad idea imho. Streaming is fine as it lowers the bandwidth demand on all players. Why do you need the aerial data of Dralnok's doom for example if your character is only L10? Why needing Aradoth right now if you are at NT for the first few levels?
Really, not everyone has 1337Mbit/s. That's why there should be both options: Streaming AND downloading, at the player's choice, where downloading is available only once per 24 hours per player to prevent bandwidth hogging + lags.

See above.

Wrong. Whether you burst load bits or stream them constantly, the amount is the same in the end.
Streaming is even better regarding those 14 days trialers who wont come back after these 14 days, as you only load the client partial instead of fully.

Download only is a -single time thing- 'burst downloading' as you call it, is very inefficient, it leads to longer loading times, more stress on the server, and more problems over all. Once a patch is released, people download the new data and are done, no more downloading.

When a player disconnects it forces the client to redownload EVERYTHING.

Also, don't assume I have a good connection. In fact, I am in a rural area with Clearwire as my provider with generally 0.9MBPS down (Before I had Digital Path with around 0.03 down), streaming is a horrible thing for me because -everything- takes ages to download by stream. One consecutive download would be a godsend for me, personally.

Streaming IS a hassle on the company, have you read about the reviews of Istaria (Horizons originally)? Ones specifically about bad lag due to loading into new areas where creatures don't appear until the player is dead? Straight download of the game would work wonders for both the clients and the company itself.

[Ettanin]

Streaming IS a hassle on the company, have you read about the reviews of Istaria (Horizons originally)? Ones specifically about bad lag due to loading into new areas where creatures don't appear until the player is dead? Straight download of the game would work wonders for both the clients and the company itself.

But would take ages for 56kers and UMTSers (I am speaking about the european area) to be able to play if it were download only.
Losing 3 to 4 days loading the entire game and it's patches, both requiring login of the 14 days trial (i.e.)? Not ideal.

That's why streaming should be an option, not mandatory.
However, streaming needs a change in the world cache systematics. And that's why i suggested to leave no limit to the WC size, because the current WC size limit is the reason why the game 'forgets' long ago visited areas.

Streaming works very well in certain other mmos.

[Dhalin]

But would take ages for 56kers and UMTSers (I am speaking about the european area) to be able to play if it were download only.
Losing 3 to 4 days loading the entire game and it's patches, both requiring login of the 14 days trial (i.e.)? Not ideal.

That's why streaming should be an option, not mandatory.
However, streaming needs a change in the world cache systematics. And that's why i suggested to leave no limit to the WC size, because the current WC size limit is the reason why the game 'forgets' long ago visited areas.

Streaming works very well in certain other mmos.

Streaming should be an option, yes, but we should change it so that the client doesn't discard terrain data until/unless the version numbers don't match, as the terrain very rarely changes 'cept for updates to the client.

Once you download it, it should be there always basically, unless it changed.

Plots/building/lair data would have to always be downloading though. Players keep changing that stuff.

[Akrion]

Wrong:
Download only: Server load burst. Lag for everyone;
Stream only: Clientside lag only for the player.
Streaming on demand is better.

Download only: Lag for everyone over a few days as everybody gets up to date, this is no different than downloading a huge revamp. Once it's done all bandwidth is free to use for anything else.
Stream only: You are absolutely incorrect about this. Streaming does not only lag the client. Your statement is physically impossible. Streaming also demands bandwidth from the server which is bandwidth the server cannot use for anybody or anything else (bandwidth dedicated solely for the client its streaming to). The more people the server must stream terrain data to, the less bandwidth the server will have available to stream any other game data with. Bandwidth usage isn't improved, it's just reassigned.

Client downloads, server uploads. Client downloads at 500kb/s, server must upload at 500kb/s. Streaming or not, it doesn't matter. If the client is recieving information, the server is sending it. How it's being sent is completely irrelevant. You don't just get free bandwidth for streaming.

So exclude low-bandwidth and bandwidth limited players as well as UMTSers? Bad idea imho. Streaming is fine as it lowers the bandwidth demand on all players.

Streaming does nothing to bandwidth demand. Usually it's even more demanding, especially on the server. Tell me, what's the difference between streaming at 500kb/s and downloading at 500kb/s?

Why do you need the aerial data of Dralnok's doom for example if your character is only L10? Why needing Aradoth right now if you are at NT for the first few levels?

Why not just stay with the current method, then, which is clearly working so well as is. That way everybody will continue to stream like it's always been done.

Really, not everyone has 1337Mbit/s. That's why there should be both options: Streaming AND downloading, at the player's choice, where downloading is available only once per 24 hours per player to prevent bandwidth hogging + lags.

You talk about "How other MMO's do it" and yet ignore the fact that most every other recent MMO has had at least one multi GB patch. Everybody has to download it. It works just fine for them. How is this in any way different?

Wrong. Whether you burst load bits or stream them constantly, the amount is the same in the end.
Streaming is even better regarding those 14 days trialers who wont come back after these 14 days, as you only load the client partial instead of fully.

Here's the problem: Everybody downloads terrain over a few days, causing a fair bit of lag for the servers until most are complete, freeing the servers of a great deal of lag. Or most people can just continue to stream the terrain, changing nothing from how it's already being done.

Streaming should be an option, yes, but we should change it so that the client doesn't discard terrain data until/unless the version numbers don't match, as the terrain very rarely changes 'cept for updates to the client.

Once you download it, it should be there always basically, unless it changed.

Plots/building/lair data would have to always be downloading though. Players keep changing that stuff.

However, this isn't really streaming. It's still downloading, it's just not downloading all the terrain at once.

[Akrion]

Question to the devs: Is Istaria even truly streaming the terrain currently? Or is the client doing something alike "Oh, I don't have the terrain data for this area, let me query the server for it." ?

[Ettanin]

Question: Should a trial player get the whole game data? I doubt that anyone would be of high level in 14 days unless one plays 24/7 which is also unlikely. So the trialer won't need late-game aerial data. Full download is unnessecary and costs very much more bandwidth for the player which could be saved if only the necessary data is downloaded and later needed data on demand over time. So only players with a subscription should get the option to download fully.

Also note that UMTSers only have a quota of up to 5 GB per month in Germany. One multi-GB-patch is essentially one full month being offline. That's not a fair and nice way to get a broader customership. Note that not everyone gets unlimited broadband, especially in rural areas.

Both options, however, should still be available: If a highspeed player is able to load the full aerial data without getting negative experience, let him. Those players can avoid later data streaming.
For low-bandwidth or limited bandwidth players, however, as i said above, a full download is not an option, it's an impairment. For those players only loading the currently necessary data is better as it's a lower bandwidth demand on them.

Also note that static terrain storage on a local basis allows certain types of maphacking (which was the way to get to the GM isle on WoW), which is imho also not ideal.

[Ettanin]

In relation to http://community.istaria.com/forum/s...ad.php?t=23658 i'd suggest the sale of an entire world cache image with a maximum age of 1 week, preferably tuesdays (as the servers are down there for mainteance).

When static download should be implemented, it's a good compensation for players with slow connections or limited download quota.

The price of a DVD/DVD pack could be USD 3.00 + shipping. I am sure that there would be some buyers if the system of how aerial data is handled gets changed.

[Amarie Ancalimon]

The suggestion for the DVD is the same essential topic as the original, thus I've merged the threads.

However, I can also answer the question that was raised and say that we can not provide a DVD copy of any portion of the game.

[Steelclaw]

Question to the devs: Is Istaria even truly streaming the terrain currently? Or is the client doing something alike "Oh, I don't have the terrain data for this area, let me query the server for it." ?

So, here's how the terrain system works:
- The terrain is broken up into 500m x 500m squares called sectors.
- Each sector has an ID number.
- Each terrain sector ID has a version number.

The client will attempt to load a terrain sector whenever the nearest edge of the sector enters max_load_dist (a function of the view distance slider). When the client attempts to load a terrain sector:
- The client will determine the ID number of the sector.
- The client will send a sector version request to the server. (increments the 'v' value in lagwindow)
- The server will send back a version number. (decrements the 'v' value in lagwindow)
- The client will check the version number against the number stored in terrain_info.def
- If the version number matches, the client loads the contents of the world cache, done.
- Otherwise the client requests new mask and heightmap data from the server (increments the 'q' value in lagwindow, once for each mask or heightmap).**
- The server will send back mask and heightmap data (decrements 'q' value in lagwindow, once for each mask or heightmap received)**.
- Once the client receives all maps and heightmaps for a sector, they are saved to the world cache, a LOD tile is created***, and the terrain_info.def file is updated with a new version. The client then displays the terrain.

** This is an asynchronous step. The server may send an unsolicited response and the client will accept it. (That's where the dynamic part comes in to play.)

*** Failure to create a proper LOD tile on disk is the biggest cause of the sort of world cache corruption that results in instant crashes upon entering an area, especially when logging in.


So, why does the client keep downloading terrain it has already cached?

In most cases, this is due to corruption of the terrain_info.def file. There's a maximum size limit (has been addressed in an internal code branch), which is less than the expected file size for a fully-populated world cache. There's also a chance of corrupting the def file when the client crashes, or when multiple processes try to write to it (multiple client instances sharing the same world cache).


Why does the terrain download take so long?

Each sector version request requires a database lookup. Each mask and heightmap request requires a database lookup. The heightmap and masks are sent down the same transport channel as the other game data; a couple orders of magnitude of size over what the transport normally handles.


As far as streaming vs static download goes, a sector download requires approximately 250K of traffic, or a little bit less than loading the main forum page twice. You guys can do the math and figure out which works better in your situation.


Those playing over UMTS and similarly capped providers do have a point, and I don't think anyone here wants to alienate them. Since these changes are a long way off from going live, there's time to investigate other implementations that can address everyone's needs.

[StalePopcorn]

Way down-under, where Internet pipes are measured by the width of the piece of string connecting our outback shed to "the local pub" (a.k.a. watering hole, etc), download speeds are on average 1MB to 5MB in cities (some higher obviously) and 256KB to 2MB in the country. Download limits are usually 10GB-20GB a month, most have free downloads in "off-peak" (12am-7am).

While I think the client obviously needs to be able to check versions on land sectors and download when it needs to, it would be hugely useful to have a distributed manual download location as well. Even if it were something as simple as a web page with download mirrors:

Land Area - Version Number - Date Updated - Size - File
Kirasanct - 1.1 - 1-Jan-2011 - 200MB - Kirasanct.zip
Dalimond - 1.2 - 2-Feb-2011 - 250MB - Dalimond.zip
EVERYTHING - n/a - n/a - 2.5GB - Everything.zip
...

This would give an option for players "in the know" (no need to keep it secret mind you - but if people want to, they could go ahead and download) to download the tiles themselves and, after placing in the [<game folder>\downloaded tiles] folder, the app could crack open the zip files, check version/validity, and unpack to their "final destination".

Just my two cents

[Akrion]

Thanks for the answer, Steelclaw. So it's partially streaming, though it's only terrain verion data and not terrain data that's in the stream.

As far as streaming vs static download goes, a sector download requires approximately 250K of traffic, or a little bit less than loading the main forum page twice. You guys can do the math and figure out which works better in your situation.


Those playing over UMTS and similarly capped providers do have a point, and I don't think anyone here wants to alienate them. Since these changes are a long way off from going live, there's time to investigate other implementations that can address everyone's needs.

Thing is, in the long run a stream demands more bandwidth. Streaming implys that terrain data is not permanent and will be replaced. Someone streaming the terrain would have to re-stream that terrain each time they passed through an area. Static one-time downloads of the areas you pass through are very different.

I think what the options should be here is an option to download all the terrain at once, and an option to download terrain as you go. Streaming terrain will do no good in my opinion.

[Steelclaw]

A properly working world cache would only require a re-download when the sector's version changes. Right now, I'll concede that the world cache, as implemented in the client, is broken.

It's discussions like the kind happening in this thread that provide design considerations for how to go about fixing it.

As far as streaming vs static download, that's going to take some programming trial and error to see what is feasible. In either case, we'll need to get the terrain data out of the client transport stream and into an out-of-band channel if it's not static.

[Ettanin]

I'd prefer the terms partial download and full download to avoid misunderstandings.

Partial download is ideal for low-bandwidth players as well as capped players and UMTSers because this mode should only download (and keep) the area nearby your character. Aerial sectors only get updated if ever visited again.

Full download is ideal for high-bandwidth players who are also not capped. In this mode, the client checks the entire aerial repository of the server and downloads it. If there is aerial data on the client, the client downloads the remaining sectors and only changed sectors.

Players can switch between modes at will.

These modes both have disadvantages:
* The aerial repository cannot run on the same server/connection as the shard itself. If doing so, lag may occur severely for everyone. On both modes.
* Local storage of aerial data is prone to maphacking. The version number can become faked local sided easily.

[Akrion]

Players can switch between modes at will.

These modes both have disadvantages:
* The aerial repository cannot run on the same server/connection as the shard itself. If doing so, lag may occur severely for everyone. On both modes.

What? Aerial repository? o.O What are you even talking about? And what do you mean it can't run on the same server?

* Local storage of aerial data is prone to maphacking. The version number can become faked local sided easily.

There's that aerial word again... Terrain data is already stored locally, it's just dynamically updated. The version number won't be as relevant when it becomes static anyway.

I really don't think you understand the concept of what would and wouldn't be easy in this regard, but even if it was easy, it would be pointless. What would faking the version number even accomplish for you, assuming you found a way to do it in the first place?

I have absolutely no idea what you're talking about or trying to say, Ettanian.


Steelclaw:

Might it be possible to provide the download in segments? You could reduce the impact upon the servers for when you make the change by releasing partial segments of the terrain data for download over the course of a month, so that every player isn't downloading everything at one time. This wouldn't really solve the issue of low bandwidth users, but it should help blunt the initial spike in stress on the server upon the change.

Like this:
Week 1 - Release the terrain data for the area around main citys.
Week 2 - Release the terrain data for the area around settlements.
Week 3 - Release the terrain data for the area around guild communities.
Week 4 - Release all remaining terrain data.

I'm just not sure if you could set some areas to be static while others remain dynamic, which this would require you to do so.

[Ettanin]

In n00b words, sry for the expression, a repository is the same as a download location/file.
Aerial in the meaning of terrain.

So the aerial repository is where you download terrain data.

As long as you don't store a copy of that 'repository' on your machine, the aerial/terrain data is server sided. You basically download terrain segments from the server in both suggested modes, only the momentarily needed bandwidth and the effective download duration (you'd take longer to get all the sector data in partial download mode as you have to visit entire Istaria) differ. You'll download ALL terrain data segments in full download mode (which is large and thus bad for low-bandwidth and/or capped players) or in parts as needed (which is better for low-bandwidth and/or capped players) in partial download mode.

Full static terrain downloads mean a long-term offline time (typically until the next start of the month) for capped players which will alienate a certain group of istarian players. German UMTS users do NOT get a cap-free download time.

What i suggest is loading the book in parts, relatively to which parts you are currently reading, instead of entirely without deleting the already read parts.

Problem with static downloading is that you need an extra server which stores the currently valid terrain data to compensate lagging by bandwidth shortage.

Also is another problem that the locally stored terrain data may be manipulated by a nefarious hacker to get somewhere where the devs do not want players to ever be. (i.E. hacking a long and straight bridge to drakul for bipeds)

The current method of getting terrain data is good, the only flaws are that it interrupts regular gameplay flow data and it's deleted periodically.

[Akrion]

In n00b words, sry for the expression, a repository is the same as a download location/file.
Aerial in the meaning of terrain.

So the aerial repository is where you download terrain data.

"In n00b words" Except I've never heard the term 'Aerial Repository' used to reference a database of terrain data, or even used in a technical meaning anywhere. Repository yes, Aerial Repository no.

A repository is simply a location of storage or a place where things are deposited. It is not 'the same as a download location/file.' To use stricter programming terms, a 'database' is where data is stored (such as terrain data) and a 'repository' is where meta-data is stored (Such as terrain version), not to say that's in any way how it's structued, but to simply use it as a basic example.

And 'Aerial' has almost no relation to terrain in any way so I still have no idea how this word could reference terrain data in any way. I mean I guess .aei could be short for 'Aerial' but it just seems like a word pulled out of random to say something other than 'terrain'. And 'Aerial repository' to say something other than 'Terrain database'.

As long as you don't store a copy of that 'repository' on your machine, the aerial/terrain data is server sided.

Istaria already does, it's your world cache. The server essentially hosts the master files which the client then uses a copy of. My console based ASCII game uses something like this to save changes to the maps for the current game while their unchanged state remains safe in reference files.

What i suggest is loading the book in parts, relatively to which parts you are currently reading, instead of entirely without deleting the already read parts.

And this is not truly streaming as you were early suggesting, which is what Starstilanx and I were initially talking about. They're just static downloads as needed.

Problem with static downloading is that you need an extra server which stores the currently valid terrain data to compensate lagging by bandwidth shortage.

You seem to not understand that the terrain data is already stored on a server, either on the individual shard servers or on a server of its own. Where do you think the client gets the terrain data from each time the cache is cleared? Either way, moving from dynamic downloads to static downloads would change absolutely nothing in this regard. So it doesn't have anything to do with it being a static download, and it should already be accounted for.

Also is another problem that the locally stored terrain data may be manipulated by a nefarious hacker to get somewhere where the devs do not want players to ever be. (i.E. hacking a long and straight bridge to drakul for bipeds)

Terrain is already locally stored as stated in a couple posts. So again nothing will change. Server holds master file, client holds local copy of said master file -> Client checks version with server -> Server updates client with new copy of *file* if versions don't match (or presumably don't exist) -> Client now holds updated version of master file.

And again I think you're severly underestimating what would be required to physically change the map data. Regardless, the danger of this happening wouldn't change from a dynamic download to a static download.

The current method of getting terrain data is good, the only flaws are that it interrupts regular gameplay flow data and it's deleted periodically.

Where have you heard that it interrupts 'regular gameplay flow data' ? The primary problem with the current method is that it causes long loading times, not that it interfers with other packets.