Debate on testing of deltas

[Chasing]

Hi All

Please note that this thread is not meant as a criticism/flaming of the devs or of the blight testers.

I am concerned that we have had a number of occasions now where bugs in deltas have been patched to live and caused problems (the most recent one being the AROP being broken by the new dragon adventure quests).

I have tried to have a think as to why this is happening (I know that there will always be some bugs that creep through). the question is, is sufficent testing being carried out before deltas are pushed to live?

The way I see it works at the moment is that a delta is pushed from genesis to blight for the players and devs to test. The patch notes are placed onto the forum so we can see what the changes are. Players can then log onto blight (if not resident) and check for any bugs and post on the forum.

However is there a specific process for blight testers to check the changes? A player can check whichever part of the patch that they want to. Therefore if no bugs are posted on the forum for a part of a delta, does this mean that it has been checked? or that no one has tested it?

My concern here is that if no bugs have been reported on the forum for part of a delta, is it assumed that everything is okay and can be patched to live?

Should we when we are testing patches to blight be posting threads saying 'yes this part works'. At present it is only intermittently whether people post to say yes to whether something works or not.

Also should there be more communication when blight patch notes are posted to say 'this has been changed, can you please check that x, y and z still works?'.

For example the AROP has been broken by the new dragon adventure quests. This could have been picked up on Blight if the question was asked 'New dragon adventure quests have been added, can you please check that the AROP still starts'. In this case my understanding is that changes were made to the AROP quest to recognise the new quests as a criteria.

Please understand I am not trying to instill bad feeling towards the devs or testers, but to provide feedback as to how we could reduce the amount of bugs reaching the live shards.

Thanks

[Mensar]

Devs test them on blight also. We had a glitch last night (bug really) that was reported on the forums and fixed in about 30 minutes flat by Lady Velea.. it was pretty amazing actually.

[Shian]

For the most part, they have been specific.

In the recent testing, the instructions were to the tune of, "We have resized plots. Please go look for any markers or stray objects that do not belong in plots."

These are great instructions and we were able to keep an eye out for these as I'm sure that while resizing plots etc. they saw that this may be problem due to whatever reasons. An example would be, The resizing was rushed, therefore these are things I know I skipped over, therefore I need to make sure it is caught.

There's also the things no one person alone thinks of. For example, the "Items must be equipped to use" change. It was several days before someone said "Hey, this affects our ROP as we usually hotkey it."

Someone once told me that developing an MMO was like building a large castle with legos. Except every time you put a piece on, two would fall off because one of them would not fit anymore and the other one changed it's shape in response to the new piece.

I'm guessing adding in new dragon quests, while not directly affecting the AROP (requirements are still the same as before the quests were put in) this is one of those times the piece changes shape and falls off for no ungodly reason.

Kinda hard to look and ask to test for something you expect to work in the first place.

[Velea]

Thank you Shian. That's a great way of putting it.

The people on Blight do an awesome job of testing. But sometimes the unexpected and unplanned does happen. Sometimes this is simply because no one person (or group of people) can think like large groups of people. The RoP example is a great example of that. If no one on Blight or the dev team who work on testing hotkeyed RoP items, why would we even think to test that?

And yes, sometimes bugs just happen. Or happen because, as much as we'd like to, the characters on Live are just different enough from those who are testing on Blight that the unplanned for happens.

I personally think that the number of actual bugs which have gone through to Live servers in the past year is minimal, and thus I applaud everyone who helps test, be it Blight player, Blight visitor or developer.

[LaughingOtter]

In all seriousness, my suggestion would be to watch the patch notes and, if you see something that would impact part of the game that is important to you, log over to Blight and test that part of the game as soon as you can. That's the best way to find & fix things before they go live.

[Mensar]

Wow, two dev responses back to back.

*holds up his "Casual Blight Visitor" T-shirt

Getting into blight is harder sometimes than others because of the difference in content (can be a sizeable download). Any suggestions to make it easier for folks to get into blight and test things other than a straight log in that requires a content download?

[Chasing]

I understand what has been said, but could you please answer my questions regarding the process? Is there room for improvement?

[Velea]

Sorry, Chasing, I thought I answered that earlier...No, to be honest, I think the only "room for improvement" is for more people to take time out of their normal play to test.

As I said, I think that the testers we have do a great job, and the bugs which make it to a Live server are few and far between.

Mensar, it is possible to have the game installed twice on the same computer. I'll see if I can't find someone on the team to write up a guide for how to do that. Then you leave one install patched to Blight and another to Live. You use a registry edit change to swap.

Edit: Actually, in thinking about this more.. there is always "room for improvement". No one and no thing is perfect. The question is, is there room for realistic improvement. And is the time invested that would be required to make that improvement worth while. So I get to modify my answer and stick with what I've said as a summary... The testing process is great as is and very few bugs make it to live.

[Chasing]

Well my suggestions are there use them if you want, if you don't then fair enough

[Zexoin]

Mensar, it is possible to have the game installed twice on the same computer. I'll see if I can't find someone on the team to write up a guide for how to do that. Then you leave one install patched to Blight and another to Live. You use a registry edit change to swap.

We have a guide in french on how to do that. I can translate if you want me to

[Creme]

I could have sworn that in one of the initial "herald" responsibilities, testing was on the list.

I found it VERY helpful (and this was some deltas ago) that a detailed list was put out of specifically what players should be looking for.

But I do think there needs to be documentation of what quests/items link to other things and have those outlined in a list of things to be tested.

It's kind of like hyperlinks. If you change a page name, you have to go through and update all the locations that hyperlink to that page. Some companies require indexing of all cross-referenced materials so that if a change is made in one document, the owner knows to update all the other documents.

[Justa Mirage]

Wow, two dev responses back to back.

*holds up his "Casual Blight Visitor" T-shirt

Getting into blight is harder sometimes than others because of the difference in content (can be a sizeable download). Any suggestions to make it easier for folks to get into blight and test things other than a straight log in that requires a content download?

What Velea said (and the french connection)

The registry entry for Istaria is not that big, a few lines and that's it.
Simply copy your entire copy of the game to a second directory, on the same drive or a different drive, and change the registry to reflect the copy you are playing. I am at work but pretty sure someone can put up pretty pictures of how to export a registry key, call it "Istaria_regular" then change the registry key to point to the new directory, export the registry key as "Istaria_blight" (both exported to desktop as .reg files)

if you want to play blight, you double click the "Istaria_blight.reg" file and then launch the game from a shortcut for the silentlauncher.exe in the new directory so it points to the new drive / folder.

If you want to play regular, just double click the "Istaria_regular.reg" file and then the silentlauncher from the original copy of the game.

[Enter]

Also, if you are using Windows XP, you can use Fast User Switching to jump back and forth between multiple Users who are all running separate instances of any program.

[StalePopcorn]

I understand what has been said, but could you please answer my questions regarding the process? Is there room for improvement?

As Chasing said in the first post, this is in now way a flame. I am still giddy with delight that we are actually getting patches, let alone stuff that works

RE improvements in the process, as the Blight server is part-Dev and part-Community testing environment, would it help if we had access to more detail about what tests have been run? I obviously don't know how testing is done internally, but some (I won't say most, because in reality, software testing is rubbish in most companies organisations have test plans and test scripts they run through to validate that a bug is fixed and that other stuff is not broken. If they are used by Devs can we see those? Might help with the testing process on Blight, because then people could:

a) see *how* a test has been run;
b) validate that a test does indeed work (we're all Human - we are designed to miss stuff
c) check if the test script does enough to validate the fix

My experience in testing is that it's the things you don't think of that break. The Virtrium team can't test everything, or think of everything - Blight testers help to fill that gap. If the testing notes/scripts/whatever were shared somewhere, this might help...

There are several reasons why this *can't* happen of course, including:
a) difficulty/time consuming/expensive providing read-only access to test notes;
b) intellectual property issues;
c) just not a good idea

My two cents worth...

[Creme]

Good post SP.

Checklists are good. Perhaps a standard checklist of things to test for different types of content would be helpful if stickied.

For example:

Quests
1. Verify mob names if specified
2. Verify locations
3. Verify requirements
4. Ensure quest log updates at each step
5. Verify completion and reward
6. Check for sub quests working correctly

Items
1. Verify description
2. Verify effect works correctly
3. Verify price is in line with other similar tier items
4. Check for conflicts/stacking of effects
5. Verify requirements to equip/use
6. Verify item STACKS in inventory ()


etc.

Can't think of other categories off the top of my head.

[Zexoin]

alright, translated it as good as I could. Guide made by Firebrand.

Installing Horizons with a registry file

What is this trick used for ? Well, it allows to install the game without using the Cds or download the intaller, if you already have an old install of the game. It's very usefull if you want to reformat your Windows hard drive and Horizons is intalled on another partition, or if you want to transfer the game files via a network connection or any other mobile storage unit (external HDD, flash drive, ect). You can also switch between two HZ installs using this trick (ie one for Blight, one for Live login).




The registry file


There are two parts in this file, one, essential, that allows Windows to see that HZ has been installed, the other, optional, that allows to create a shortcut in the add/remove programs section of the control panel.


Step 1 : Open the notepad (start => run => type "notepad" )


Step 2 : copy/paste the following :



Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\ArtifactEntertainment]
"PatchVersion"=hex:05
"PatchDidcaspol"=hex:01

[HKEY_LOCAL_MACHINE\SOFTWARE\ArtifactEntertainment\ Horizons]
@="Path of the Horizons directory"
"NewPatchVersion"="11"
"NewPatchDidcaspol"="10"

[HKEY_LOCAL_MACHINE\SOFTWARE\ArtifactEntertainment\ Horizons\version]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Curr entVersion\Uninstall\{0BCE91C3-5AF7-43E1-A071-9C550B76D6D7}]
"UninstallString"="RunDll32 C:\\PROGRA~1\\FICHIE~1\\INSTAL~1\\PROFES~1\\RunTim e\\0701\\Intel32\\Ctor.dll,LaunchSetup \"C:\\Program Files\\InstallShield Installation Information\\{0BCE91C3-5AF7-43E1-A071-9C550B76D6D7}\\SETUP.EXE\" -l0x9 "
"DisplayName"="Horizons - Empire of Istaria"
"LogFile"="C:\\Program Files\\InstallShield Installation Information\\{0BCE91C3-5AF7-43E1-A071-9C550B76D6D7}\\setup.ilg"
"ProductGuid"="{0BCE91C3-5AF7-43E1-A071-9C550B76D6D7}"
"InstallLocation"="Path of the Horizons directory"
"DisplayVersion"="1.00.000"
"Version"=dword:01000000
"MajorVersion"=dword:00000001
"MinorVersion"=dword:00000000
"LogMode"=dword:00000001




Of course you already know that we'll have to change what's in red... replace it by the Horizons directory path, but be careful, you must use double anti-slashes in the path to get it to work.
Example : if your directory is C:\Program Files\Artifact Entertainement\Horizons, you must type C:\\Program Files\\Artifact Entertainement\\Horizons


NB: the path must be the new install one!!!


Another note : if your windows install is not on the C: drive, don't forget to modify the reg file to point to the right partition.
exemple: C:\\Program Files\\InstallShield Installation Information\\{0BCE91C3-5AF7-43E1-A071-9C550B76D6D7}\\setup.ilg


Step 3 : Save as a registry file


Go in File => Save as... then choose the file type => all, give the file a name followed by the .reg extension.
Exemple: Horizons Intall.reg


The registry file is ready to be used.


Step 4 : "install" the game


all you need to do is to double-click on the registry file you've just made. Accept when you're asked if you want to add the infos to the registry.


And voilÃ** !

[Firebrand]

Guide made by Firebrand.

It was a translation of a post from Fridlekh that must be burried somewhere in this forum. Credits go to him.

[StalePopcorn]

Good list Creme - could we maybe also add:

QUESTS
- Verify that area-driven step works (i.e. you arrive in a location that kicks off a quest entry)

ALL
- Checking spelling/grammar (I know, I know, it's very petty, but people notice these things. Or at least I do. )

I would imagine that testing on Blight is a bit limited in some cases - we (the players) obviously can't type in a magic code and have our character magically change to level 100 in order to test something. But all the points you raise should be possible.

Testing is big. And it's hard. And you rarely get everything right. There are so many combinations that it quickly get's impossible to test all the outcomes. But most items the players can quickly verify.

[Guaran]

As far as getting ready to help test on blight, and easily moving back and forth between the live shards, I will offer up how I do those things:

1) Get yourself copied to blight for testing. This is really the most necessary step. This might take a week or 2 to happen. Get a high level dragon and a high level biped copied over. (whatever you have)

2) Set up batch file logins for all your characters on both live and blight shards. (not that difficult and a time saver once it is all done)

3) When you need to login to blight to test something, or when you need to patch up/down if a delta gets moved to live (this is usually right after maintenance), use the horizons stand alone launcher or website to login. This patches you up to current blight status.

4) "I though you said this was easy?!" This is the easy part. After testing on blight, you can usually login to live without patching down. Just use the batch files for your live shard characters. You may in fact be pre-patched for the next live delta. I never assume that tho, and after a live shard maintenance, I login to live once using the stand alone launcher. Nine times out of 10 I can move back and forth between live and blight with no patching at all once I'm patched to the blight client. This is where testing made easy happens. This usually works since most of the time, changes that need tested are in fact server side (backend) changes. It has never happened to me so far that the blight client is incompatible or breaks some function on live shards. Some sort of massive client update would have to be in the pipe for the to possibly happen.

Once you batch files are in place, simply make a few shortcuts to them for each character-shard. I also keep a stand alone launcher shortcut handy.

I used to use 2 different installs of the game but after they added the new client where you could view customizations on linked items, I wanted to try it on live. Turns out its easier just logging into live with the blight client and not patching down, to minimize changes (editing regitry or renaming directories as I used to do it) and time spent patching.

If you do go the reg-key route, the uninstall string isn't used for gameplay and can be skipped.
From zex's post:

[HKEY_LOCAL_MACHINE\SOFTWARE\ArtifactEntertainment\ Horizons]
@="Path of the Horizons directory"

This is the only change you need to make to toggle between different locations. Don't forget to double slash.

[Takora Drakan]

I could have sworn that in one of the initial "herald" responsibilities, testing was on the list.

FAILED! Sorry for being grumpy but people don't seem to read annoucements!

Annoucement regarding Heralds and Emissarys: http://community.istaria.com/forum/s...ad.php?t=17841

* Emissaries help to test for bugs in new content, verify bug reports, and summarize player feedback about the game through frequent reports on an internal forum set up for Emissaries. While it is not required that Emissaries play on the Blight shard, it is expected that Emissaries have at least one copy of their character on the Blight shard that can be used for testing purposes.

*Sighs and wanders off*