State of Minkata (long version)

[rarified]

It's been a long time -- far too long -- since I last took an opportunity to talk to you.

Today, I'd like to give what I'm calling a "State of Minkata" report. More than the bland homilies of "I'm working on it" or "Soon(tm)".

Unfortunately to do that I'm going to have to do a deeper dive into both history and technical detail. If you need to don Maintainer suits to keep your heads from exploding please do so now.

For the first part, I'm going to go back in time, to when Cyan decided to release the Uru game client as open source. This was, if my aging noggin recalls, back in 2011. Prior to that point in time Uru had gone through a tumultuous period, first the original release, then the closing, then the resurrection, and once again a closing.

During that (brief) time Uru and the Myst franchise had already established an amazingly resilient fan community, both technical and not. During Uru's downtime, a lot of community effort went into reverse engineering how Uru worked, much of it as I understand it with the intention of making Uru still available to those who wanted to participate in it. I was not part of the Uru community during that time, but there were underground "shards" available, with varying degrees of success in reproducing the Uru environment.

The point here is that a pool of technical expertise grew during that time period, with knowledge of the inner workings of how the Uru game client and server worked.

Forward to around 2011, when Cyan committed to releasing the Uru game client in open source form. This is when I started to become involved in a technical sense, and was invited to join OpenUru.

One of the factors that I think influenced Cyan's decision was the appearance at that time of a new, open source game server designed by two of the community members a'moaca' and cjkelly1. My understanding is that they had been for quite some time part of the technical group interested in reverse engineering the Uru system. And at the time Cyan was considering releasing the game client, they were ready to release their game server called MOSS.

When the commitments were finalized from both the part of Cyan and the MOSS team, I volunteered to create a shard called "Minkata" that I would manage. It's goal was to provide a "sandbox" (hence "Minkata") for testing changes that the fan community wanted to contribute before proposing to Cyan that their changes be made part of the game. Minkata would be running the MOSS server for it's shard.

MOSS was an extraordinary piece of work. The authors were seasoned computer software professionals, experts in the computer languages of C++ and SQL. The server was comprised of almost 60000 lines of C++ code, plus several thousand more lines of SQL for databases.

There was one quirk, however, in my hosting Minkata. I too was a computer software professional, at the time employed by Sun Microsystems. Sun had
a flagship product, a Unix operating system called Solaris. Unix was the precursor to Linux, designed at Bell Laboratories in the 1970s,
and eventually very successful in the commercial world. Unfortunately, it never was used by the public communities because it was hideously
expensive, and took a great deal of expertise to install and manage.

I ran a server at home, that was built on Solaris. The same server I was going to offer Minkata from.

By the time of the open source Uru client, Linux had started to establish itself as a replacement for most if not all commercial Unix installations. It was free to individuals, and was becoming closer and closer to an interchangeable system to Unix.

MOSS was developed under Linux.

What this meant was that in order to run MOSS on Solaris, I would have to install all of the software that MOSS depended on. Software is now never something that stands alone when an author finishes his code. He or she depends on code that others have already written so that they do not have to "re-invent the wheel". Think of the "Russian doll" toy. Open up one doll and you find another nested inside of it.

For me, the inner dolls existed, but only in source code form. I would have to compile and install all of the dependencies that MOSS used.

I was happy to do that, but it did take time. I was able to complete the task by the time we made the Uru source code available, and Minkata
was born, running on my Solaris server.

We spent the first year, ironing out how to make MOSS and Minkata function smoothly. Some of you may remember the "stress test" parties
during that year where we crammed almost 150 players onto Minkata before it broke in a way I needed time to fix! But we made progress, both with the stability of Minkata, as well as starting to get contributions to the Uru game client.

In the time since then, the amazing part has been that MOSS has required zero updates and bug fixes, to keep Minkata running. That is extraordinary in the computer software world, let alone the open source software world. It is a tribute to the thought that went into the original MOSS implementation.

During that time, unfortunately, a'moaca' and cjkelly1 have withdrawn from active participation in the MOUL world. It is a significant loss, as you will see in a moment. In the meantime, Minkata kept chugging and contributions -- game client contributions -- made their way back into MOULa.

Also, during that time, the server running Solaris also just kept working. Sun was bought by Oracle, I retired from Oracle, and my server never needed updating. I only had to reboot it when I had power failures that lasted longer than my battery backup. I also never needed to change any of the software that MOSS depended on.

Until last year.

Last year, actually the year before that, my server experienced a security breach. It wasn't surprising, the secure code used to access my server was years old, and hackers had that time to discover flaws in the code they could exploit to get in. This meant that I would have to start updating underlying code libraries that implemented security on my server.

The authors of those libraries were not idle in the years I had not changed anything. Linux was evolving, and things that those libraries depended upon, even the compilers, had been updated. Sometimes so significantly that the new versions would not work with code built with the older versions. So my adaptation of MOSS on Solaris had to be rebuilt, along with the newer version of code it depended upon.

This meant that among other things, the next time I had to rebuild MOSS for Minkata, it would be using the new underlying libraries.

Also during last year, a'moaca' and cjkelly1 reappeared with some updates to MOSS. They too had observed that MOSS no longer completely worked with contemporary versions of Linux, and had developed code fixes to make it compatable with things like newer encryption software and a new version of the Postgres database which was the current "standard" version being distributed with current Linux.

And last year I started working in earnest on how to get Fan Content into the pipeline for testing and eventual inclusion on MOULa. I had planned to add a new shared I've been calling Minkata-prime on the same server, for testing fan content. To run two shards on the same server required some changes to MOSS.

So MOSS needed to be rebuilt.

I completed that task late last fall, and thought I had successfully migrated Minkata to a new MOSS server. Unfortunately feedback soon appeared that things were not quite right on Minkata. In particular, sometimes things that were changed in ages during game play reverted to their prior state when an explorer returned to that age. This is what is termed "state", which has to be stored by the game server (i.e. MOSS). And it was failing after the rebuild.

So Minkata has been malfunctioning since October of last year.

On and off since then until January I've been poking at it, but with only a small amount of time available for work on it.

Now for the current "State of Minkata".

Since the beginning of the year I have been able to spend a fair amount of time trying to narrow down the possible causes for the Minkata bug. During this time I've disabled all previously existing accounts on Minkata, which makes it much easier to debug things with a new Vault. Minkata is still locked down today.

I've eliminated the database as the cause of the problem, and identified a part of the problem that MOSS is not creating or writing to a file when it is required to save state in the game. The task now is to determine the cause for skipping the file update. Unfortunately the original code for MOSS did not include substantial tools for diagnosing this type of problem. So I have been writing additional debugging code and working to incorporate it into MOSS to give me more insight into how the logic of that particular functionality is working when a player in the game causes a state change. Right now, I've modified close to 1000 lines of code in order to implement those changes, and have not yet rebuilt MOSS with the new code. I am optimistic that once
I am able to run Minkata on the updated MOSS I will be able to identify the cause of the problem.

So, given that state, what is the path forward?

First, Minkata needs to be made whole again. I am continuing to work on it, now with a fairly consistent schedule of time each week. I don't know what the problem is yet, so I can't give a schedule on a fix. I do know I have a few more days of work before I can test the updated MOSS.

Second, start testing and adding changes to bring Minkata-prime up coexisting with Minkata. Again, I know there are changes needed, and I've had to read code both in MOSS and the Uru game client (CWE) to understand where the changes need to be made. Without that information, I can't give a time frame, but expect it to take a couple of weeks.

Third, I have some fan created content, both very simple ages for proof-of-concept as well as an older copy of Doobes' Pub. I was able to experiment for a short time last year with these age additions, and found that the game client crashed or malfunctioned with them. So it is likely that there will need to be bug fixes added to the game client before we can experiment with fan content. Once I am at this point I will enlist the aid of the GoW members, opening Minkata-prime to them. Hoikas has already mentioned a possible bug that was fixed in H'uru that may be a factor. I've also heard that the Blender module for producing Uru ages has been updated several times since I did my first experiment.

So in conclusion, I'm saying that while you have not heard anything from me about Minkata and fan content for a long time, there has been considerable work being done to reach the point where we can deploy it in a testing framework. And once that barrier has been broken, the community can start working with Cyan to finish the bridge to MOULa

Thank you.

_R

[Treehugger]

Thanks for the explanation, rarified, and for your huge amount of work It's greatly appreciated! I hope you are soon able to complete this marathon enterprise!

[Emor D'ni Lap]

Rarified, thanks so much for the detailed rundown on the history and rationale behind the Minkata shard's setup.
I can't thank you enough for all the work you've put into this effort over the years!

Now, I hope you don't mind my asking a few questions, and I want to be sure you know I'm asking entirely out of ignorance: I am not a sysadmin or even much of a coder. And I do understand that you set up the server using Solaris because that's what you have and that's what you know; I get it.

Still, I have to start from the point where we agree that the Minkata shard's whole purpose is to act as a one-to-one-equivalent testing ground for Cyan's shard. So, I have to wonder whether the difficulties you're experiencing getting the square Linux peg to fit the round Solaris hole are causing Minkata to drift further and further away from serving its original intent. Changing over 1000 lines of code, etc. ...does this get us closer? Is content that runs properly under MOSS guaranteed to run properly on Cyan's server? I'm certainly not qualified to say.

But, I do wonder whether we would be better off with a server that modeled Cyan's own shard technology in as many respects as possible?
I have no idea whether you'd be comfortable as an admin of such tech, nor do I know whether we'd be able to financially support such an approach...but I guarantee that I would chip in a good share.

And the other positive aspect to doing this would be that some of the responsibility for the server could be delegated among a few trusted people in our (cough) *community*?
I know I'm not the only one who feels badly that you've had to shoulder this load solo for so long.

[rarified]

Hi and Shorah, Emor!

Thanks as always for your comments.

I'm going to take your 2nd last question first, since it bears on the majority of my posting.

While I never have spoken to Chogan, Rand or Tony directly about the Cyan MOULa server code, I have always been under the impression that Cyan has no interest in making it public. I know of some pragmatic coding reasons for this, and can speculate on other logistical reasons. So it never was considered an option to build a non-Cyan shard based on Cyan code, open sourced or not.

So we as the community have to fall back on the earlier work done with reverse engineering the protocol, as well as (now) having the CWE client code to confirm or disprove the RE analysis, as well as give additional insights into how a server would work.

The advantage the MOSS server brought to the table was that it was, as far as I could tell, an implementation of "what is", in contrast to "what we want it to be". It was the fidelity to the existing ecosystem that was attractive, and that it worked with all of the MOULa features right from the start.

Ok, that being said, to the topic of the added code and the Solaris environment.

I think I look at running MOSS on Solaris as a benefit to portability, rather than a downside. I have tried very hard, when finding coding that didn't work under Solaris, to determine whether the problem was with an irreconcilable difference between Linux and Solaris semantics, the use of a Linux extension or perhaps bending of the C/C++ standard behavior, or (horror) a simple mistake in how a standard interface was used. There have been maybe two or three instances of semantic differences that required additional code, and that has been (unfortunately) integrated with conditional compilation (i.e. #ifdefs). But most of the "fixes" I have implemented will work identically on Linux as on Solaris.

I have been remiss in this regard since many of these fixes I deemed local to Minkata on Solaris should go back to the MOSS general repository, but are not there now. They only exist in a private fork of the MOSS code I have for Minkata. After my TODO list shrinks, that will move back up on the priority list.

However, to the mention of about 1000 lines of changes. That was computed from a difference from the Minkata-prime working copy of the code against the public repository. It includes probably a couple hundred of the aforementioned fixes, and the rest are code additions to assist debugging the current Minkata problem. They don't change (or are at least not intended to change) the behavior of MOSS in comparision with the Cyan MOULa server.

The bulk of this debugging code is changes to add code to produce marshalled string representations from Class objects, with the addition of methods to produce C and C++ string values describing the contents of those objects. I'm particularly focused on data structures describing game age state, also known as SDLs.

This is not going to change the behavior of MOSS as compared to the MOULa server. Preserving that behavior is paramount for Minkata.

Now for the TL;DR part:

For example, as part of the game data, there are files that describe the state of the age Relto (Personal in MOUL-speak). In those files, we find a descriptive line for a variable that keeps track of whether the door to the hut is closed:

Code: Select all

    VAR BOOL    psnlLibraryDoorClosed[1]    DEFAULT=1

As part of tracking down why the door doesn't stay open (along with all the other states of ages that are not being preserved right now), I need to locate where that state is being stored, both while a player is connected to the game server, as well as after they have left the age and perhaps logged out.

That state is maintained in MOSS game server memory while a player is present in an age through a pair of C++ classes called an SDLDesc{} (describes the attributes of the state) and an SDLState{} (which describes the current value of that state). When a player is not present in that age, the contents of the SDLState{} are saved to disk storage in a file. The SDLDesc does not change when an age state changes, so an individual copy of the description is not needed.

When a player returns to that age, the game server looks for a file with the saved state for that player, and re-converts it into another in-memory SDLState{} object for the rest of the server to use.

My additions to SDLDesc{} and SDLState{} classes provide methods to construct a string representation of the contents of an object of that class type. For example, when the descriptive SDL file with the line above is read by the game server, the resulting SDLDesc{} object string would look like this:

Code: Select all

"psnlLibraryDoorClosed<BOOL>(true)"

This string literal could then be logged to a debugging log file to verify at a particular point in SDL processing that it was correctly interpreted.

After I visited the hut and then left Relto without opening the door, the SDLState{} string representation would be

Code: Select all

"psnlLibraryDoorClosed<BOOL>(true): true <SameAsDefault|HasMaximumValue>"

and that value would be written to the state file on disk.

If I revisit the hut and open the door, the SDLState{} (read from the state file) string representation becomes

Code: Select all

"psnlLibraryDoorClosed<BOOL>(true): false  [Sun Feb  3 16:26:22 2019 1549236382.689769]  <HasTimeStamp|HasDirtyFlag|WantTimeStamp|HasMaximumValue>"

With a tool that came with MOSS, and modified to use my routines, I can examine the state file and display the information above. In this way I have been able to confirm that the change of state has been properly written to the state file, and that the bug causing state to disappear happens when restoring the age state in the game. Which I did last Sunday when the final changes to the SDL classes were completed.

So there you have it -- the last information gleaned in the efforts to diagnose the "Minkata age state bug."

_R

[Christian Walther]

So it never was considered an option to build a non-Cyan shard based on Cyan code, open sourced or not.

Add to that that Cyan’s server runs on Windows, and you’d probably find as many volunteers to administrate a Windows server as for Solaris.

(Also, was MOSS really developed on Linux? I seem to remember it was NetBSD, but can’t find a reference right now. It also worked fine on macOS (another BSD) last time I tried, a long time ago, so it seems pretty portable.)

Is content that runs properly under MOSS guaranteed to run properly on Cyan's server?

It is relatively likely because the server doesn’t actually have all that much to do with the content (with the exception of the SDL). The content doesn’t so much “run on” the server as on the client. The server mainly routes messages between clients.

It’s worth emphasizing that the problems Minkata is having right now are totally separate from the question of fan ages. The problem with fan ages (or specifically, with Doobes’ Pub, which has been the only fan age submitted for MOULa, to my knowledge) is compatibility with the client. Last time I checked, anyway. And the Minkata client is very close to the MOULa client.

[cjkelly1]

Also, was MOSS really developed on Linux? I seem to remember it was NetBSD, but can’t find a reference right now.

It was developed on both. I did the SQL database stuff on Linux, and a'moaca' wrote the server code on BSD. We tested on both BSD and Linux, to ensure there were no issues.

When I had heard of the Minkata state issues, I checked to see if those issues existed on my long running server, and they did not. I then loaded a fresh Ubuntu Server 18.04 and compiled MOSS from latest source. I was again unable to reproduce the problem. Unfortunately, I do not have sufficient C/C++ skills to be able to offer any other help in that area

[rarified]

After today’s debugging time, it appears MOSS has some memory management issues.

Running a newly built MOSS server tonight (with the alterations to the SDL classes and logging code) yielded a functional server that properly restored the age states. I have no reason to believe anything I did “fixed” anything. Instead, I suspect an out of bounds memory access is lurking somewhere.

I’m going to eyeball the SDL code some more before trying to plot a course of action. It may turn out that reenabling public access to Minkata would result in either discovery of other incorrect behavior, or a server crash I can work with.

_R

[Emor D'ni Lap]

Rarified, thanks for the education on MOSS/Solaris and the specific nature of the code alterations, appreciated.

So glad the server is seemingly behaving now; I put in some testing time yesterday and found no issues at all...but then I didn't run through all of Ahnonay, for instance

As previously mentioned, if in your testing you have any use for a very simple Age created through the 3DSMax/Plasma exporter, please let me know. I'll be happy to add specific features you'd like to test, over time, as needed and within my limited capabilities!