Multiplayer: the part of Falcon that generates the most queries from new pilots and experienced hands alike. Most of us now are familiar with the basics of getting connected to other players over the net or in a LAN.
This piece will examine what happens "under the covers" of Falcon SP3 and SP4 during MP sessions. It blows away a few myths surrounding the tricky issue of bubbles and explains what messages are divided up between the host and the client (s). It is the most comprehensive guide to date on the workings of MP in Falcon.
The article is compiled from a myriad of messages over the last week or so from various sources which have proved most informative. If you're not sure about the basics of Falcon MP, I recommend reading my "MP More" sticky in the Falcon 4 Forum over at frugalsworld. Also see the guide reported by Bruce "Lytning" Hale at http://home.ptd.net/~bhale/BandwidthTable.htm
Over the last 18 months or so, two people have really shaped the way MP works in the SuperPak versions of Falcon: "RIK" and "Schumi". Their work changed the way in which MP operated by modifying the code in the exe. It's complicated stuff and tricky to test but has generally greatly improved the Falcon MP experience.
Delving into Multiplayer
Back to Basics
Before getting into the hub of the piece, a few definitions:
HOST: The machine which puts up the game (not necessarily the one which has 0.0.0.0 in the internet line)
CLIENT: The machine(s) which connects to the host's game.
ENTITY: anything existing in the Falcon 3D world, such as a building, a tank, a missile launcher, plane, SAM site, etc.
DEAGGREGATION: the process where an object in the 3D world becomes a fully-featured and operating entity. If every object in Falcon is deaggregated, we would need a couple of CRAYs to run it because of the number-crunching. So every player, be they host or client, will have to deaggregate an object in the 3D world, just as they do in any offline mission.
BUBBLE: Imaginary spheres around any entity in the 3D world. When the player's plane moves into an entity's bubble, that entity deaggregates and becomes proper 3D entity. It's the method Falcon uses to save CPU cycles.
In order for MP to work, we need to send messages between the various players taking part. In other words, each computer needs to know information about where the other player (s) is in the 3D world. This is done by sending messages between all the PCs in the MP game. Those messages include updates on the position of each plane, as well as updates on the position of units such as a SAM site or a missile in the air and so on.
AI missiles controlled by Host
Hosts, Clients and Peers
Falcon operates a client-server system for multiplayer. Peer-to-peer was the old pre-SP method of operating a multiplayer environment. The trouble was it could be wasteful of bandwidth, causing unnecessary bottlenecks in the system. Hence the move to client-server code to make things more efficient. The host in Falcon controls the positional updates for all AI planes. When an AI aircraft's position changes, these changes are sent out to the clients to let them know those new positions.
Taking Control of Entities
F4UT has done tests using network sniffer/analyser software to track how data flows between the PCs in a MP game. It has found that data such as positional updates of AI aircraft are sent using a client-server method, ie the host sends updates to the clients. As stated earlier, the host controls all AI planes, hence why the host experiences a big FPS drop during intense A-A engagements with AI planes -- the host is carrying out the bubble calculations for those aircraft.
Will otherwise known as "Wmulvihilldxr" -- has done a considerable amount of work testing and tracing on the MP side to verify what happens. He concludes:
1. When the client joins the game, the host sends the client a BUNCH of "Create/Full Update" events. These events are pretty darn big and if you are in a campaign, you'll need to receive one of these events for every ground unit, squadron, and plane in the campaign. NOT objectives. These full update events also give you the exact locations of each of these entities as well as which PC owns and controls the unit (remember this last fact for later). As long as you are using the same campaign, your client session will simply load the OBJECTIVES from your local machine. How you find out about damage comes later...
2. Thus, the client has a copy of every single entity in the entire game (objectives, planes, ground units, etc, etc.) to start with and knows exactly where each one is.
3. At this point, the host owns all of these entities.
4. If an objective is damaged, you don't find out about it unless you need to. Why would you need to? Well if you used the Recon window in the UI to view an objective, you'll get a full update on that objective including any damage to things.
5. When someone joins the 3D world, they get to the second pie and will deaggregate any ground units in the area of the airbase.
6. But wait, you thought the host owned everything before? But when a client deaggregates the entity, it takes ownership of the entity from the host and then deaggregates the individual units within that entity. For example, if you deagged a Patriot battery, it would deagg the individual launchers and support vehicles. The client owns all of these.
7. At this point the client has this ground unit that it owns. But it also tells the host that it deagged the entity. The host then becomes a SLAVE to the MASTER (the MASTER being the Owner, i.e., the client). If the host needs an update on the entity, it will ask the client.
8. So this is how clients receive control of entities and from then on (until they aggregate the entity), they control it.
10. NOW, a second client joins the game. As before, the host will send the 2nd client all the full updates on the entities.
11. Then the second client joins the 3D world alongside the first client.
12. What happens now? Well remember that the host still has a fully updated copy of all of the entities. It just happens that the first client owns that Patriot battery one. But if the SAM needed to be updates, the 1st client would have done the calculations and then told the host about it.
13. So when the second client gets into the 3D world, it still requests the deaggregation of the Patriot Battery from the host.
14. The host says, "Hey, I've got that entity. Its already deaggregated and I don't own it, here is the deaggregated information, enjoy."
15. And the second client now has the entity and knows that it is owned by someone else (besides the host) so this second client DOES NOT take ownership of the entity.
16. However, if the second client flies away from the first and is the first one to deaggregate a different entity, then this second client will take ownership of this different entity as usual.
17. Now what if both clients are sitting in the same bubble and the Patriot battery that the first client owns moves somewhere.
18. That would trigger a position update to be generated on the 1st client's machine. Why? Because it owns the entity, it will do the AI and decide what it does (don't worry, its not as simple as "All the clients get to have ground units running around like idiots." There are some checks in place to make sure the units work in a manner consistent to the overall war. Its not entirely integrated, but it does help this).
19. CRITICAL STEP. Here is the answer to the question that seems to be clouded. When the first client generates a position update for that Patriot Battery, it does NOT send it to the 2nd client in the SP3 code. It sends its update to the host. The host recevies this update and in its own copy of that entity, marks it as "dirty" Being marked as dirty means that the host knows its changed and that the next time the host gets around to sending position updates, it needs to propagate that new update to other clients that need that update.
So, the reason that clients can own entities and yet still not have a ton of bandwidth sucked out and also the reason that the host needs a bunch of bandwidth to support more clients is that the Host actually does give position updates to all the clients. However, its the clients that do the deaggregation and control all the calculations necessary to manage that entity. The clients pass that information on to the host and then the host can (if needed) pass that information on to the other clients.
Bottom line:
1. The clients take the CPU hit for deaggregating and managing entities.
2. The clients pass that information onto the host.
3. Thus the host has the most complete picture of the game at all times.
4. The host supplies position updates to all the other clients if they need them (i.e., if the clients are in the same bubble as the other clients or request deaggregation of that entity).
5. The host takes the bandwidth hit for #4 by having to propogate the entities out to everyone that needs them.
What about the "Host All Units" option in Falcon? Wasn't the idea of that to take the heat off the clients by assuming control of all ground units in all players' bubbles?
Host All Units option
Well yes, that was the idea, but it seems to have worked less efficiently than one might of hoped for initially. More testing is required, but it should be pointed out that CTD problems in MP are often due to bandwidth restrictions on the network.
Compression: the name of the game
But there is also more magic in Falcon. It uses a clever compression technique to save bandwidth by cutting down on the number of packets (individual data streams) sent. For example, a dial-up user who deaggregates a ground object needs to send his friend flying as his wing a positional update about that unit. Not only that, he may well also have to send a positional update of his own position...and vica-versa. It would seem for each that would require two packets of information to be sent (unit position + aircraft position). But Falcon has a trick up its sleeve. Whenever a ground unit update needs to be transferred, that piece of information is simply tacked onto the regular plane position update that happens every half a second or so. Two bits of information for the price of one. Nice. Each positional update is around 48 bytes. For a dial-up modem, each packet sent can contain something like 30 positional updates -- an efficient use of bandwidth.
Actually, this trick of appending messages together in a packet not only happens in positional updates, but it happens with all packets being added to the sending queues. So if you've got a bunch of different packets that are all going to the same destination, Falcon will keep appending them together until either the packet gets sent, or the packet gets too big to fit across a dial-up connections MTU (MTU basically means the maximum number of bytes that can be transfered in one network packet).
If the first packet gets too big, Falcon just starts another packet and keeps appending to that. It is common to see Falcon packets with 2 or 3 different message types in the packet. When the game is fast and furious in 3D, you can bet on each network packet containing an average of 5 to10 seperate messages inside. A fantastic way to squeeze every byte out of the network.
One thing to clarify about compression. Not only does Falcon append messages together to use one network packet for a bunch of messages as described above, but, in addition, when the packet is getting ready to be sent out over the network, it is compressed using a widely known compression algorithm. Thus a 1500 byte packet, might turn into a 800-1200 byte packet. That also saves bandwidth.
Spreading the message
Of course there is much more data swishing around a MP game: for example if you spike your human wingman, messages have to be sent to achieve the warning tone in the wingman's pit. The list goes on and on. Below is a detailed explanation of what happens.
Falcon also has a number of ways to deliver the message from a networking standpoint. A little networking background first. You may have heard of TCP and perhaps you've also heard of UDP. The advantage of TCP versus UDP is that TCP has a number of built-in protocol features that ensure that your network packets will reach their destination intact, in order, reliably, and without corruption. The disadvantage to TCP is that it makes your network packets bigger to implement all of these features. UDP trades reliability and order to gain more bytes back for the packet data. That is, it uses up less bytes in the packets so that you can fit more data in the packet.
Falcon sends packets in 2 different ways at the network level. UDP and RUDP (Reliable UDP). UDP is connectionless (versus TCP which can maintain a constant connection between 2 PCs) and assumes the packets reaches its destination. The minimum and maximum size of a UDP header is 8 bytes. RUDP is an extension of UDP and it does what it says it does, its Reliable UDP. It provides many of the same features as TCP but it uses UDP as the transport (meaning RUDP is wrapped in a regular UDP packet). RUDP is a semi-proprietary protocol. Cisco, I believe, has an RFC (RFC=Request for Information. Basically RFCs are the documents that standize network protocol for use on the Internet and beyond) on their RUDP protocol, but it looks nothing like what is in Falcon. But that's getting off track.
RUDP headers (the bytes at the start of a network protocol within the network packet) in Falcon have sequence numbers, ACKs, fragmented messages, etc like TCP to ensure that data gets there intact and is acknowledged by the receiving side. But here's the difference, RUDP headers will take up no more than 9 bytes. In most cases, it uses only 5 bytes for the header. So what does that mean? Well with TCP, according to the RFCs, you absolutely have to have AT LEAST 20 bytes in the header, sometimes up to 192 bytes. With RUDP, depending on the type of RUDP packet, you have from 9-17 bytes in the header (8 bytes for the regular UDP header, and then 1-9 bytes for the RUDP header). In most cases, you'll have about 13 bytes in the total header versus TCP's required 20 bytes.
What about the data integrity that TCP gives you? Well Falcon has multiple size fields in the actual data sent with the packets to ensure that you aren't going to CTD by receiving some bad data. Plus, both IP and UDP (and TCP of course) have checksum fields that will ensure that the data was delivered correctly by the network interface (your ethernet card or modem or whatever).
So in the end with RUDP, you have the data integrity, reliable transporting, and acknowledgements of TCP, but with 3-11 bytes savings on every single packet. And in fact, RUDP is a little looser in its implementation of ACKs where instead of acknowledging every single packet in a row, you can acknowledge a bunch of packets at once simply by giving the last received sequence number in your ACK. This tells the other side that you successfully received all packets up to that last received sequence number and can mark those packets as delivered successfully. So you save even more bandwidth with that.
What is the bad part of RUDP? In general, RUDP is not as robust and tested as a TCP implementation. But it does its job pretty well.
This brings us back to the beginning. We know Falcon has 2 ways of sending packets, UDP and RUDP. Now there is a delicate balancing act here on deciding what messages should be UDP and which should be RUDP. You don't want to have everything be RUDP because of the extra bandwidth involved in it. But you don't want critical messages being lost on the Internet becuase you just sent it regular UDP. For example, the game generates Radio Chatter messages (not regular radioing back and forth between wingman and the like) like when someone on your team (not necessarily in your package) dies you hear that "ahhhhhhhhhhhhhhhh" messages. Those kinds of messages that are not critical to the continuation of the MP game are sent UDP. If you get that radio message, great! But if you don't, your MP game isn't going to be adversely affected by you not hearing that sound. On the flip side, you really need to know if a missile you fired hit the target and record the damage accordingly. That needs to be RUDP. But there are special cases involved in choosing RUDP versus UDP too. For example, position updates are sent so fast and furious that making them RUDP would delay the processing because of the need to ACK the packets. By the time you ACKed an RUDP position update, it might be obsolete and the next 2 position updates are fresher. So position updates are sent UDP not because they aren't important (they are!) but because of the latency involved.
And on top of all that, there is another key balancing act involved in Falcon MP. For each type of protocol (RUDP and UDP), there is the choice of sending stuff immediately or putting it in a queue. So you have UDP now, UDP queued, RUDP now, and RUDP queued. "Now" messages are also refered to as "OOB" (Out of Band) messages because not only are they sent as soon as possible, they also ignore the available bandwidth, meaning, they do not go through the normal checks to see if bandwidth is available.
Here are some examples:
UDP now -- Position updates. You want them sent immediately and unreliably like I explained
UDP queued -- Some connection controls stuff that is not that important and can be put off
RUDP queued -- Most messages fall into this category. They need to reliably get to the PCs but aren't immediately needed.
RUDP now -- A few messages need to be sent immediately and reliably, like missile tracking messages (you are being tracked by a missile basically)
So there is more of a balancing act between 4 types of packet delivery methods. Out of the 80 or so types of messages in Falcon, about 75% are sent RUDP. That's a rough estimate. Position updates are the most used and they are definitely sent "UDP now." But actually position updates are a special case. They have a number of different checks in place to make sure the bandwidth isn't going to be exceeded by a lot. So really, position updates ARE sent immediately, but their bandwidth usage is checked at the higher lever in the code. Its a complicated matter and one that would go beyond the scope here.
I hope this piece has explained in more detail what happens during a Falcon MP game, and where bandwidth is allocated and in what circumstances. It is a complicated area and one which causes a huge degree of confusion. The beauty of Falcon is that its code was way ahead of its time. The idea of bubbles is just making its way into First Person Shooters now. And by all accounts the Unreal MP engine is not unlike the type that runs Falcon.
My thanks to the expertise of Will, who helped me considerably with this article. Also thanks go to Schumi and RIK, whose coding efforts for SP3 gave us what we have today.
