Improve Uru Cost/Performance Using UDP Network Architecture

[Nalates]

@Admin, thanks for the split. Keeping things on topic will, I suspect, make this a great forum for researching issues.

@Mac, I think you are right as to lag (server & network). I have had heard that from a couple of other people I trust to have hands on info.

SL provides some interesting debugging tools built into the viewer. So, while I cannot personally measure MOUL performance for SL I can watch the fps rate and the actual bandwidth use as well as see the texture cue and other 'items' to download remaining. Plus loads of rendering information. There are literally screens and screens full of stuff to look at. It is easy to find performance bottle necks.

Often when my performance degrades I look to see what has happened. Once I've downloaded everything for a given area its my client that is struggling.

For now the only place I expect milliseconds in network traffic to matter is in server side communications, server to server. If we do spread servers among various homes, we may find a bottle neck.

If we add more abilities we might see more server load. More moving things would mean more traffic. But, what I gather from what Mac_Fife writes and a couple on GoW the pipeline is not yet full. It seems that will mean MOOS would need to change quite a bit before there would be real value in changing network protocol. I can't see possibly incurring the added work and a future operation cost.

@Ned, I'll go with a'moaca on wanting to see how UDP gets a better routing.

And there is a difference between listening and believing. Extraordinary claims require extraordinary proof, which is why I'm waiting to see if RealXtend and Blue Mars type virtual worlds really can put 1,000 players in a ('a' as in a single) server and not have lag. I've listened, I am skeptical and I won't be putting time/money in either until I see it in real life use. You are in the 'skeptical' items slot too.

Most people don't have the time to test out new tech and ideas. I have to look to see what the big kids are doing. I make the assumption, right or wrong, that larger well funded companies do research into anything new. If they are not switching, their research likely did not show enough advantage to make a change worthwhile. Often it is a matter of people needing to see a tech/idea in use before they begin adopting it.

Telling anyone they are not listening or are closed minded, is not going to open minds. Politicians do that. For many only hard facts and rational explanations will change their beliefs. Once one resorts to talking about personal behaviors, the debate is over and it is more like political propaganda.

[cjkelly1]

No. But tweaking the network protocol may help to avoid teleporting avatars.

This would seem to be a client issue as well. Since the protocol is TCP, I will get all the packets and I will get them in the correct sequence. If the client is too busy to process the received data at the time it is received, and then tries to catch up later, how could this be fixed by changing the protocol?

I believe Mac_Fife has the correct assessment. There will be more gained by resolving the client issues than would be gained by modifying the network protocol. Since these issues have existed since the UbiSoft days (when Uru was using UDP, by the way), I do not believe they will be all that simple to fix. Were it simply a matter of tweaking the network protocol, I would think Cyan would have done so by now.

[Inquisitor]

Maybe I'm reading some of this wrong...but here goes anyway.

I'm not a programmer or a sysadmin by training. I'm from the school of hard knocks and dirty hands learning. And I don't know the real differences between TCP & UDP as far as speed and dependability are concerned. I do understand the client side lag problem though. And I agree that it's in direct relation to how powerful a given system is.

What I'm trying to say is this. I have a ProLiant DL580 G2. Maybe not the latest and greatest of datacenter class servers...but it does what I want it to. Ned seems to be saying that people like me with bigiron servers at home should just put'em on a shelf and go his route? I don't agree with his premise. I have reliable information that tells me, with the server I have, that I can run a medium size "shard" with a fast cable connection. I'm unsure how many will be able to play at once on a medium size shard. I do know that I had as many as 50 people on my old UU shard at one time. Admittedly there was considerable "lag" but they could still move. That was on a home PC with less than a quarter of the DL580's power, less than eighth of the ram, a single sata hard drive, and a far slower connection than I have now. So I'm going to stay optimistic as far as being able to run a shard from my home.

And I doubt that anyone would be willing to foot the bill for hosted servers when we have so many talented people in the community willing to provide what is needed for free.

[realXCV]

No. But tweaking the network protocol may help to avoid teleporting avatars.

This would seem to be a client issue as well. Since the protocol is TCP, I will get all the packets and I will get them in the correct sequence. If the client is too busy to process the received data at the time it is received, and then tries to catch up later, how could this be fixed by changing the protocol?

It can have something to do with the client but it can be network related too. If your client freeze and you can't move, it's a client issue. However, if you can move relatively easily but others avatars are teleporting, it looks more like a network issue.

[rarified]

I'm going to take a different tack here (which might be considered off-topic)...

I would like to understand what problem the OP is trying to address.

In his first message, he heads down a path of addressing how to host URU backend servers, with an assumption that they must be offered from a hosting service. I'm not sure we can assume this as yet; none of the fans have had an MOUL-class server to run and observe. All we can speculate with is observations of client behavior, and the hints that Chogon has provided that the MOUL servers have a different architecture than the UU and shard servers that fans have had experience with.

But also within that first message, he then asserts that in order to be practical the MOUL servers must run with a UDP transport architecture. c.f.

MMO LIVE gaming requires very careful UDP package management.

and

This model has been thoroughly tested and is still running today....
10 mg guaranteed output per user with streamed udp packets
built-in client udp manager (most important part)-(the colonels secret recipe).

We already have an existence proof that MOUL can and did operate with it's current protocol stack.

Finally, he makes an attractive offer to host URU services in a facility that is tailored in some way to expediently deliver UDP services for game clients.

In my mind's eye, this is a problem (as a'moaca' and Mac have observed) that is not in the critical path of restoring the Cavern. The critical path I see is:

• Accept the conditions Cyan may place on using the code
• Obtain the client and server code
• Build the client and server code+++
• Formulate a procedure for anyone to deploy the code
• Attempt to deploy the code in a sample environment to validate the prior point

(+++ - This may be unnecessary if (as I'm starting to suspect) Cyan is intending to release a ready-to-run binary bundle along with the source code)

At that point "market forces" will start asserting itself and those interested will start running servers (joining an already existing shard or branching their own.) Only after that point (where fans can once again run a client and enter the cavern) will alterations to the original code base be a practicality.

I think I can understand where Ned is coming from -- he mentions a couple of times FPS games, where reaction time (and prop delay) become a significant part of the gameplay. UDP does have some overhead advantages over TCP in that regard (although it comes with a different set of baggage). And I wouldn't discourage an investigation into the benefits of different transports. It just isn't something that needs solving... yet.

As a matter of fact, I think I'll add another suggestion thread on something related to this. Perhaps it would help if we deploy a client with profiling and performance gathering capabilities. Let me think how to word such a proposal.

[realXCV]

At that point "market forces" will start asserting itself and those interested will start running servers (joining an already existing shard or branching their own.) Only after that point (where fans can once again run a client and enter the cavern) will alterations to the original code base be a practicality.

Changes that globally affect the gameplay should however be done as early as possible so when the game starts branching, thoses changes would be applied on every shard.

I understand however that it might not be possible to do all of that before the game start branching.

[Ned]

At that point "market forces" will start asserting itself and those interested will start running servers (joining an already existing shard or branching their own.) Only after that point (where fans can once again run a client and enter the cavern) will alterations to the original code base be a practicality.

Changes that globally affect the gameplay should however be done as early as possible so when the game starts branching, thoses changes would be applied on every shard.

I understand however that it might not be possible to do all of that before the game start branching.

I agree here TCP or UDP is a big choice and indeed if there were to be a change made it should be at onset.
TCP or UDP I will still offer testing servers...doesn't matter this is not the point of discussion.

From my research and you'll have to trust me....me and my team did allot...
Some of the problems encountered with the GameTap launch was encountered due to TCP overheads.
Others in the game community chuckled over the attempt thinking it was oddly brave.

One example...
A TCP Packet is somewhat bigger than UDP.
Two avatars standing next to each other in realtime.
Avatar 1 pulls a lever to activate a game sctipt.
The activation script requires a server call to acknowledge a change in the environment.
Yet over to avatar 2....
Server sees avatar 1 + 2 movement, clothing, so on.
Activate function call is made....server makes reply...then sets new varaiables...reply again with new conditions.
meanwhile avatar 2 is hovering in lag land waiting he has done nothing but his client is being heavilly updated, or alternatively the server becomes overworked by many such instances and locks up.

Hence why most mmo games and fps games use UDP and are mostly static worlds.
Interactive environments can be achieved but the overheads are costly, so usually mmo game developers will build with these aspects in mind, considering how many avatars, how many interactions, time for server to recognize interactions and the impact the interaction will have on the environment once noted.

So back to quoted message....
Sure these changes will need to be started early.
I propose two models are tested if we are to do this the right way.
Throw up the original TCP model...
Then next to that an edited UDP model.
and bash test them both.
to determine strengths and weaknesses

from there a wise decision can be made
and game world...shard, development can begin.

Another comment.....
This is something you should all know...

Rumours around the game dev community indicate there is a quiet race on between game companies.
Not "who is first with the code", but rather "who is first to launch a successful engine" wins the community.

To game dev companies...this is money.
1000 users are told "servers are up....plug in your old worlds to ours..."
and the game company just walked away with a pre-made game.

My response...
We want to avoid this if we are to keep it open source
and although I too am a game company....I am in this project for URU's sake, not my game company's.

I am sure this has already crossed Cyans minds hence their remarks about how there may still be a centralized server...
and good for them...do they really want to just give the game away?

So to points at hand:
1/ ok run two models for testing UDP and TCP for comparison
2/ Speed is of the essence if we wish to keep it open source and free for the community.

[teedyo]

No. But tweaking the network protocol may help to avoid teleporting avatars.

The common network related issues like teleporting/warping, rubber-banding, and treadmilling occur regardless of which protocol is being used. How often and to what extent they may be manifested depends heavily on what the game's network handling does to mitigate the circumstances which result in these effects. To use your example of teleporting/warping: it is inherent in UDP and occurs due to a loss of packets, while in TCP; it is usually the result of mitigation where the client throws away extremely aged packets and continues from a 'fresher' point.

One of the benefits of UDP is that it doesn't "play fair". On a congested link; TCP backs off so that at least somebody's packets will get through; UDP doesn't care; it just keeps trying to shove stuff on the wire. Of course, in severely congested situations; this is really no benefit at all as you may have linked to Relto and nobody except you may know it.

Tweaking can be done with TCP. UDP would require a completely new application layer. This would, however, allow the designers to selectively decide which packets absolutely must get through and those which can be lost/folded/spindled/mutilated without pain. This could result in a markedly more efficient networking method. Would it be worth it in the long run? Maybe.

[teedyo]

Apologies for the double post...

One example...
A TCP Packet is somewhat bigger than UDP.
Two avatars standing next to each other in realtime.
Avatar 1 pulls a lever to activate a game sctipt.
The activation script requires a server call to acknowledge a change in the environment.
Yet over to avatar 2....
Server sees avatar 1 + 2 movement, clothing, so on.
Activate function call is made....server makes reply...then sets new varaiables...reply again with new conditions.
meanwhile avatar 2 is hovering in lag land waiting he has done nothing but his client is being heavilly updated, or alternatively the server becomes overworked by many such instances and locks up.

Whether you realize it or not; you just made a case for TCP because state variables are something that absolutely must be reliably transmitted in URU. Take the following two scenarios where the state variables for the lever and door are in packet 2:

TCP
Avatar 1 pulls the lever, a door opens, and he walks through.
Server sends packets 1, 2, 3, 4, 5
Client receives packets 1, 3, 4, 5 oops
Client NAKs packet 2
Server resends packet 2 which is received successfully. Yay!
Avatar 2 sees this: avatar 1 grabs the lever, pause, avatar 1 pulls the lever, a door opens, and he walks through.

UDP(unmitigated)
Avatar 1 pulls the lever, a door opens, and he walks through.
Server sends packets 1, 2, 3, 4, 5
Client receives packets 1, 3, 4, 5 oops goes unnoticed
Avatar 2 sees this: avatar one grabs the lever(doesn't move it) and walks through a closed door.

The problem with UDP becomes one of how to specify which packets must be received.

[cjkelly1]

From my research and you'll have to trust me....me and my team did allot...
Some of the problems encountered with the GameTap launch was encountered due to TCP overheads.

Those whom you are trying to convince are knowledgeable about these matters, and I do not think they will accept the "trust me" approach. You will get much better results if you simply post the data to back up your statements (so that others can verify them).

Speed is of the essence if we wish to keep it open source and free for the community.

Please explain this statement. How will someone come in and make OSMO non-free? If Cyan chooses the correct licensing model, this will not be able to legally happen. What is the reason that speed is of the essence?