10hr Maintenance... I Got This!

Player question: Points of caution in server management and operations
51:10
Because FFXI has been running for 20 years now… I imagine that server management and operations must be difficult.
51:18
What are you mindful of during normal operations… from an infrastructure engineering point of view?
51:26
"Infrastructure engineering" isn't a word you hear every day. I wonder if this person works in the same industry.
51:35
You have already said a few things about this. Can you give us a bit more detail? Now that 20 years have passed…
51:43
the number of customers doesn't fluctuate very much anymore.
51:49
The load on the servers and networks is essentially constant.
51:55
Despite that, even when there are no events running…
52:02
sometimes the data we are monitoring fluctuates. The graphs might deviate or move up and down in a jagged fashion.
52:08
That makes us suspect something is happening behind the scenes. It makes us intensely curious, because that's our job.
52:15
We check in with the development team… and ask if there's anything going on… or if they have made changes to the application.
52:24
If that doesn't identify the reason… then we really know that something is suspicious. We start an investigation…
52:30
looking for broken equipment or abnormalities in any of the networks.
52:37
And we do that continuously. On top of that, there is more variety now…
52:44
in terms of players' computers and environments. Some people have super fast connections like NURO.
52:52
Some apartment buildings have a shared internet connection. And then there's people who play over Wi-Fi.
52:59
Keeping the playing field as even as possible for everyone… is one of your important jobs.
53:04
Let me think.
FFXI was designed to be playable even on ISDN
53:12
The volume of data traffic must be different now… considering the changes in technology and devices.
53:19
Did that mean you had to alter the applications as well?
53:26
The original concept for FFXI was to build it to be playable over ISDN.
53:35
So in the context of modern internet usage… it uses an trifling volume of data…
53:42
compared to simply watching a video on YouTube. The game is designed to run properly…
53:49
even using the traffic on the very edges of the network. FFXI just quietly hums along in the background.
53:56
The original creators really built something amazing.
54:02
It was made by an A-team.
54:07
It's fully functional as a system, as a game, even now.
54:13
The team that made those foundations was truly amazing. I have a question for Mr. Fujito.
54:19
The circumstances and environment have changed… and even the applications have been altered bit by bit. It's been two decades of various changes.
54:27
Do you have any opinions about connectivity? I'm not an engineer.
54:34
So I can only answered based on what I've heard or felt.
54:40
Say that the game creators are introducing a new system.
54:45
We might increase the number of areas. We've made a lot more "layer areas" for private play.
54:56
These are like spaces where everyone can play their own MORPG. We have a system for doing that.
55:03
When we discuss building systems like that… it's like adding more soup stock. We get more and more systems.
55:09
Even just at the level of the game servers… more and more data is being interfaced all over the place.
55:16
When we are working on how to govern this… Information Systems is partially involved…
55:22
but it is mostly designed by the game engineers.
55:27
Information Systems handle the external aspects perfectly.
55:32
And the game engineers focus on maintaining the content of the game.
55:38
The interface between the two is where we can talk to each other.
55:44
We might tell them things could get unsteady… because we inserted a new system.
55:53
And they'll say, "What have you messed with this time?" Information Systems doesn't just handle data connections.
Not just server operations, the duties of Information Systems
56:02
They also strongly support us like the IT department in other companies.
56:10
With machine trouble, for example. They even help with that? Or if I want to test something that my current machine can't handle.
56:21
Or if I want another PC for security reasons. If you ask them, they bring a PC straightaway.
56:29
They support us like that. Then with the livestreams… just seeing Nakata's face makes me feel safe and sound.
56:36
I know he can handle anything.
56:43
People like me who just play the game… have no idea about the weight on Mr. Nakata and his team's shoulders.
56:50
But listening to you speak now… I can see that we'd be lost without you. That's how crucial your role is.
56:56
Every team needs to assign someone to do these tasks.
57:01
And instead of having to reinvent the wheel… we are supported by a battle-hardened team with a wealth of experience.
57:11
It sounds like you don't just work on FINAL FANTASY games… but you have to work on anything and everything.
57:17
That must be tough.
57:23
Here's an even more interesting question for you. Has anything particularly difficult happened recently?
A massive 2-year project, renewing servers through virtualization
57:29
It's never easy. There was something about two years ago.
57:38
Fujito and Matsui announced in the media… that the game would continue for another three years.
57:45
In reality, behind the scenes… we had a server renewal plan running in parallel.
57:51
We were planning to use virtualization technology… to run 20-year old applications on the most modern hardware.
58:01
That would reduce the number of physical machines… and as a result the size of our data centers and running costs.
58:11
Our plan combined the best of everything. We powered through the plan without pause…
58:18
and managed to pull it off in about two years, prep time included. That was the toughest episode we've faced in FFXI recently.
58:25
Does virtualization actually let you shrink the data volume?
58:33
Virtualization is a bit of a buzzword in the IT industry.
58:39
I hear about it a lot, but I'm lost. There's two key points.
58:45
Modern hardware is too powerful and too big.
58:51
One OS can't use up all the resources of the machine. So what do we do?
58:57
We use a virtualization application… that splits up the capacity of one machine…
59:02
so it can be used by multiple OSs. That means each OS can fully use the limited capacity it is given.
59:10
We can have three or four fully operational OSs on one machine.
59:17
That's one benefit of virtualization.
59:23
Also, it lets you run old applications on the latest hardware.
59:30
By adding a virtualization layer? For instance, FFXI is a 20-year old application.
59:36
Was it designed for the modern hardware coming 20 years later? Of course not.
59:42
No one could predict how technology would evolve. Hard disks have been replaced by SSD.
59:50
Everything used to be externally attached. We used to talk about 2 or 3 MB of memory.
59:55
Having a hard disk with a few dozen MB made you a hero! We've gone from MB to GB, and now we're at TB.
1:00:03
We can bridge that gap by using virtualization applications… and run FFXI from 20 years ago on the latest hardware.
1:00:14
I assumed you could easily play small, old games on modern machines.
1:00:20
But that's not the case. The fact is that old programs only run on old machines. So you need an extra layer to make it work.
1:00:28
I never knew that. I'm learning new things.
1:00:34
You said it took about two years to prepare. Did you have to replace servers?
1:00:40
I'm sure there would be issues like technology, and wear and tear. To replace a server, do you shift everything to a different server…
1:00:45
and then replace the old one? This time we fully created the new environment first.
1:00:54
Then we migrated it over bit by bit. We migrated the applications over in sections from old to new.
1:01:01
In one maintenance window, we might just move one World. We migrated gradually over multiple maintenance windows.
1:01:09
Once the old server is finally empty, you dispose of it. And replace it physically with a new machine.
1:01:17
I guess that makes sense. Of course you couldn't use the same equipment for 20 whole years.
1:01:22
It wouldn’t be future proof. The machines might already be at their limit.
1:01:30
That's why we have really small but frequent maintenance windows. We never go offline for long periods like three days.
1:01:40
It's like a standard periodic version upgrade… but just two or three hours longer.
1:01:47
We make changes in little slices here and there… and we can rollback if any issues occur.
1:01:53
So I think players probably didn't even realize what was going on.
1:01:58
We were just talking about servers. Servers aren't just a single block sitting somewhere.
1:02:06
They're built of multiple components. There must have been changes over these 20 years.
20 years of changing technology around servers
1:02:11
Granted that changing the servers themselves is one thing. What else has changed?
1:02:17
The technology around servers. The advent of virtualization as a technology.
1:02:24
That probably contributed the most… in enabling us to somehow keep FFXI running.
1:02:29
And from the point of view of the Information Systems team… when there were issues in the past… we had to go to the data center and fiddle with the servers directly.
1:02:39
Troubleshooting was done on-site in the past. But maintenance technology has evolved considerably.
1:02:48
Tasks like switching servers on and off can now be done from home… with the simple push of a button using the modern tools.
1:02:54
Operations and maintenance work are vastly easier than they used to be.
1:03:02
Online games were still in their infancy 20 years ago. It makes sense.
1:03:09
We were super impressed with ADSL. And I think fiber optic is fast.
1:03:15
But we're talking about 2003. Broadband was just getting popular at the time.
1:03:21
That's when they started calling it broadband. I thought that was more recent than 2003.
1:03:28
The connections have changed multiple times since then. And different connections mean instantly different data volumes. 1367 01:03:35,550 --> 01:03:37,000
1:03:37
But the game itself hasn't changed. It must have been hard to maintain the balance.
1:03:42
Constantly releasing version updates sounds exhausting.
Almost no content has been discontinued in 20 years
1:03:53
You've told us about the process overall. Is there anything that you are extra cautious with?
1:03:59
What do you think is the most difficult? For the whole 20 years of FFXI…
1:04:08
almost no content has been discontinued. It's just constantly increasing then?
1:04:19
Everything that we've made is still playable. The flipside of that is that…
1:04:24
maintaining these servers and applications… in their original state is like preserving important cultural assets.
1:04:35
It's like a conservation project. And we're still releasing version upgrades and additional content.
1:04:45
I think it's astounding. A lot has changed, including significant changes to security.
Information Systems also manages security
1:04:53
I feel like we're required to do more than just have a password now. There have been various changes.
1:04:59
Implementing those functions would require updates, right? Is that within your scope of work, Mr. Nakata?
1:05:06
Half of it is our job. And then there's another team within Information Systems.
1:05:14
That team works on the SQUARE ENIX ACCOUNT system. I've got a SQUARE ENIX ACCOUNT too.
1:05:19
So that's a different team? Yes, it's a separate application development unit.
1:05:26
When it comes to security… that team has very sensitive feelers out to catch new information.
1:05:32
They continuously incorporate the latest security trends…
1:05:37
into their countermeasures. In terms of security and the SQUARE ENIX ACCOUNT login…
1:05:45
I think one-time passwords rapidly became more common because of FFXI.
1:05:54
There's a device like a keyholder that you push to get the password. I remember losing mine and being devastated.
1:06:00
It reappeared when I was moving house… and I was like, "That's where you were!" It is amazing when you think about it.
1:06:06
Compared to simply typing in and saving your password all the time…
1:06:12
there are significantly fewer cases of people's accounts being hijacked.
1:06:18
It's an amazing system. Conversely, has anything remained the same over these 20 years?
The FFXI application has remained unchanged for 20 years
1:06:25
I'm repeating myself… but the FFXI application itself is the same.
1:06:31
That's the most significant constant. In contrast, trends in the surrounding IT industry are always moving.
1:06:43
For instance, you can't install an old OS on a new machine anymore. Left as is, FFXI would have become unplayable on newer hardware.
1:06:52
We had to bridge that gap with other new technology… to keep and continue that legacy
1:06:58
We work with a mixture of unchanging and changing things.
1:07:09
It is amazing that something from 20 years ago is still alive and well. They really did make something wonderful.
1:07:16
You could say that the job of Information Systems today… is to keep the surrounding turbulence under control…
1:07:23
to preserve the game in its original condition.
1:07:29
There's a trend of "not needing to change" nowadays. You still show clearly what your stance is for FFXI moving forward.
1:07:38
Let's go to a player question. How do you decide the upper limits for networks and servers?
Player question: The upper limits on networks and servers
1:07:46
I would assume that higher limits are preferable… but smaller limits are better for maintaining stability.
1:07:52
I'm interested in how you decided the limits. Is this really a question from a player?
1:07:57
It sounds like someone in the industry. It's a question about upper limits.
1:08:02
I assume this player is asking about the number of players per World.
1:08:10
I don't know for sure. There's no way of knowing anymore…
1:08:16
what criteria the initial team members based the initial decisions on. I'm not sure about the overall limits…
1:08:22
but each area is contained in a server… and the servers are managed individually.
1:08:28
So we do set a maximum number of players that can enter an area.
1:08:34
If it would go above that maximum, you can't enter the area.
1:08:41
For example, in the Treasures of Aht Urhgan expansion… there's a system called Besieged.
1:08:47
Everyone gathers in one area… and protects the city from being overrun by the attacking monsters.
1:08:56
Inevitably people concentrate in one place. Sometimes people couldn't join in even though they wanted to.
1:09:04
That's because we had those upper limits. If you let unlimited people in, it could seize up, right?
1:09:10
Yes, because it eats up more memory. Think about how player data sits on top.
1:09:17
Memory is limited, not only on the PlayStation 2. Servers can handle a wider range…
1:09:23
but you still need to set limits for the servers. Otherwise, if there's a memory overload… the server could crash and the game could stop.
Player question: The reason for lag despite the minimal data traffic
1:09:33
Let's go to the next question. This is half a request, half a simple question.
1:09:41
FFXI started in a mostly ADSL environment 20 years ago.
1:09:47
Now that fiber-optic and CATV connections are common… how has the data traffic for FFXI changed?
1:09:54
Some of my friends have said they're surprised… because they would expect an older game to have less traffic…
1:10:02
but popular content still feels laggy and heavy. Please tell me if there are current issues around that…
1:10:08
and if you have any plans to improve the situation. This is becoming like a shareholder meeting.
1:10:15
Offhand, I would have thought older content would have less traffic.
1:10:21
So I would expect it to run lighter and not be laggy.
1:10:27
That's my preconception. You're right. It's playable on ISDN, let alone ADSL.
1:10:35
The traffic is small. However, for popular content…
1:10:41
I think the bottleneck is in the server processing. It's not about the connection or your computer's specs.
1:10:48
It's closer to the source. The Asura Server is unexpectedly popular right now.
1:10:55
Because unanticipated numbers of people are gathering… it inevitably causes congestion.
1:11:02
In terms of the data traffic… in comparison to recent data-rich games…
1:11:07
the data volumes are really modest and trifling -- slender even. So that shouldn't really be having any impact, in my opinion.
1:11:18
There is also a huge abundance of content on the internet lately.
1:11:23
This is weighing down the internet overall… which I think is also having a certain impact.
1:11:29
There's internet connections for apartment buildings… where the fee is included in the rent.
1:11:35
I lived in an apartment like that. I always had a great connection during the day.
1:11:43
It was more than enough for playing games when I didn't have work. But now a lot of people work from home during the day. There are connection issues on the user side…
1:11:55
as well as server-side issues. Being a data-light game doesn't necessarily solve everything.
1:12:00
I realize that now. Speaking of which… my home internet connection has sluggish moments too.
1:12:08
There are factors beyond even the servers or the game. It's different for everyone, depending on their environment.
1:12:16
So I hope that you can all stay calm and patient with us.
1:12:23
Thank you. Does ISDN still exist? I don't know.
1:12:33
They've announced that it will be wrapping up soon. We still have ADSL, right?
1:12:40
It's still alive and kicking. There used to be a box - a separate router - on the line.
1:12:47
That's gone away now. It's a simpler connection.
1:12:52
I didn't trust Wi-Fi in the slightest at the beginning. But I find it so convenient now. Even though the environment is everchanging…
1:12:58
our ability to play the game is unchanged. And that's really wonderful.
1:13:04
I hope you all enjoy adventuring together. Is there anything else that hasn't changed? The game systems, I guess.
1:13:15
The server configuration is also completely unchanged.
1:13:23
We have "Worlds". And there are special servers that bundle the Worlds together.
1:13:29
The design is identical to 20 years ago. It's completely untouched.
1:13:36
The fact that it's still playable… means they really took great care…
1:13:41
to build an excellent system from the beginning. Without a doubt, the game is still fully operational today…
1:13:46
precisely because of the robust foundations.