Improve Ffxiah Point System To Infer Missions?

Language: JP EN DE FR
2010-09-08
New Items
users online
Forum » FFXI » General » Improve ffxiah point system to infer missions?
Improve ffxiah point system to infer missions?
First Page 2
 Siren.Codegen
Offline
Server: Siren
Game: FFXI
user: Genome
Posts: 180
By Siren.Codegen 2020-11-29 01:25:17
Link | Quote | Reply
 
So linkshell community beta has been dead for years and is never coming back. That means that newer characters or characters that were renamed by a server merge will never have points counted, because while guildworks sends items and jobs data, it doesn't send missions.
But why not infer it based on final item rewards from a mission?
For eample anybody that has Moonshade earring clearly has finished WoTG missions.

It's been like this forever but for example my character was renamed from
https://www.ffxiah.com/player/Diabolos/Genome
last logged in 2010 and with missions, to
https://www.ffxiah.com/player/Siren/Codegen
first logged in a couple of years ago and forced to pick a new name on that server, but it's still the same character, but with no missions so no points or ranking.

Seems like inferring mission from items so more people can be ranked would keep more people engaged and checking this site...

Can anybody that runs and maintains ffxiah add this?
[+]
Administrator
Offline
Posts: 6495
By Rooks 2020-11-29 09:19:36
Link | Quote | Reply
 
A version of this is what I was working on over the holiday break. There are some things I don't have ways to divine using the current system, which means changing the GW plugin (effectively rewriting it).

I've probably made three or four real attempts to do this over the years; it's not an unknown problem. The trick is finding a reliable way to do it that doesn't grind the site to a halt every hour as achievements process :)
[+]
 Siren.Codegen
Offline
Server: Siren
Game: FFXI
user: Genome
Posts: 180
By Siren.Codegen 2020-11-29 19:04:46
Link | Quote | Reply
 
maybe you could ask the player to wear all those gear items that prove they finished those missions and do a gw flush and logout, now they show up on their profile as last worn.
Then you could add 1 button on ffxiah site that says "validate missions from gear", so it only runs once on demand. When it runs, it updates the info with the max between existing, lowest possible value and inferred from gear, so if mission was empty, it becomes either 1 or 10 if they have that final reward gear. Even being given a rank with some points missed from missions might be better than no rank at all.

Another way to infer it might be if you have access to the new mastery rank from gw, for example I am rank 6 and I think it must be because i did all possible missions among other things.
Offline
Posts: 42635
By Jetackuu 2020-11-29 19:13:05
Link | Quote | Reply
 
Eh, would almost have to have a separate process that processes the data then just reading that from the site (could even just fully utilize gw for this I guess?), or you know just update at jp midnight (slightly cruel). idk, I don't envy the concept, definitely a task.

Knocking out methods to pull xyz data isn't even half the battle, to do it reliably and so it's not trying to pull from so many people at once is the trick. Makes sense, as the LS community beta was a good buffer, same with the gw plugin (and even it hangs often).
 Asura.Arico
Offline
Server: Asura
Game: FFXI
user: Tename
Posts: 535
By Asura.Arico 2020-11-29 20:05:12
Link | Quote | Reply
 
Rooks said: »
I've probably made three or four real attempts to do this over the years; it's not an unknown problem. The trick is finding a reliable way to do it that doesn't grind the site to a halt every hour as achievements process :)

How is GW able to update gear/REMAs in realish-time without crashing the site?
Administrator
Offline
Posts: 6495
By Rooks 2020-11-29 21:46:02
Link | Quote | Reply
 
Asura.Arico said: »
How is GW able to update gear/REMAs in realish-time without crashing the site?
I don't want to get too far into the weeds here, mostly because it's not done yet and I may find a problem that requires a real change to the methods. I have a mostly working prototype that I'm doing performance testing on, so this is almost certainly reflective of what will make its way to production, but things change etc etc etc.

So, right now, the GW update process for REMA/etc is done on the fly, because there's basically no thinking to do. Item list comes in, I check it against a list of IDs, and off we go. There's no calculations of any kind. Additionally, when missions were handled by the LSCOM, it was a similar scenario: we got a rank number or a quest ID, flagged it appropriately, and moved on.

Both very simple, basically no thinking, at most one or two things to check at a time, as GW sends a delta of items, not the whole list.

What I have to do now, is a bunch of those checks. No particular one is hard, but they're all a little diffferent: I have to call up people's titles, check their whole list of items to see if they have a Rajas Ring, etc. I don't want to potentially do all of this math every time someone hits //gw flush or zones, so hourly with the achievement processor is the correct answer.

What I have done is create more traditional achievements - got WAR 99, mastered BRD, etc, complete with silly names. When you //gw flush, your character record will be marked as changed, and when the achievement parser runs it'll recalculate you, specifically only calculating for achievements you don't currently have, e.g., once I flag you as having completed CoP, I never have to check for a Rajas Ring again. That's the best balance I have between "I have to do a bunch of math now" and "I'd rather not peg the CPU every day during prime time as two thousand Asurans flush data every time they zone".

I'm currently (as in, before I started writing this, and will be getting back to it after) fixing titles to be properly synced between GW and AH. They're not right now. Both of the sites are missing some and they're missing different ones and the IDs don't always match. It wasn't anything I did, so this has likely been broken for years.

I also need to come up with more silly names for achievements.

Make sense? Questions? Again, still a work in progress, but I hope to have something for Scragg's approval this week.
[+]
 
Offline
Posts:
By 2020-11-29 21:47:16
 Undelete | Edit  | Link | Quote | Reply
 
Post deleted by User.
 Siren.Codegen
Offline
Server: Siren
Game: FFXI
user: Genome
Posts: 180
By Siren.Codegen 2020-11-29 22:07:54
Link | Quote | Reply
 
there's also not 20k characters doing brand new missions at once either. If you started with characters that have blank info for missions but have jobs info, that would be another way to phase them, and again that would be a computation that would have to be run at most just once per character, since right after the first time their missions would no longer be blank.

After that works, you can then worry about handling the case of updating mission info for people that have it and don't have it capped, and you can choose to run that infrequently etc.

The first step that has no chance of overloading resources as it's at most a once per char computation (zero for many that already have missions info) would already let all those accounts start to be ranked, and the others would at most be a little lower than the correct value if new missions were not counted.
Administrator
Offline
Posts: 6495
By Rooks 2020-11-29 22:13:15
Link | Quote | Reply
 
Siren.Codegen said: »
no chance of overloading resources as it's at most a once per char computation
This is why I say:

Rooks said: »
specifically only calculating for achievements you don't currently have, e.g., once I flag you as having completed CoP, I never have to check for a Rajas Ring again

Since titles/items/key items can all be reduced to lists of IDs, it's really not an issue to grab the list of what a player has once an hour (assuming they've even flushed data to GW), sift out anything interesting, and flag as needed. It'll honestly be less strain on the system than what achievements does now.
 Asura.Arico
Offline
Server: Asura
Game: FFXI
user: Tename
Posts: 535
By Asura.Arico 2020-11-29 22:30:57
Link | Quote | Reply
 
Gotcha. I misunderstood what you were saying. Finding a reliable way without rewriting guildwork.

I was thinking that it wouldn't be very difficult to get a delta of quest/mission IDs, but that requires a rewrite of guildwork.
Administrator
Offline
Posts: 6495
By Rooks 2020-11-29 22:35:06
Link | Quote | Reply
 
Asura.Arico said: »
I was thinking that it wouldn't be very difficult to get a delta of quest/mission IDs, but that requires a rewrite of guildwork.
Specifically a rewrite of the client, which I've started. The problem is that there's a big chunk of functionality that's annoying to replicate, but is used by premium users, and sees real use. I can't replace the client until I replicate that, and with my free time being what it is right now, doing the processing AH-server side is the quicker fix for now (even if it won't give a complete picture).

Eventually it all needs replaced, but eventually the sun is gonna go nova so we'll see which one happens first I suppose
 Siren.Codegen
Offline
Server: Siren
Game: FFXI
user: Genome
Posts: 180
By Siren.Codegen 2020-11-29 23:53:21
Link | Quote | Reply
 
Rooks said: »
Siren.Codegen said: »
no chance of overloading resources as it's at most a once per char computation
This is why I say:

Rooks said: »
specifically only calculating for achievements you don't currently have, e.g., once I flag you as having completed CoP, I never have to check for a Rajas Ring again

Right, and I understand you already coded part of your approach, I was just suggesting an even easier intermediate option if that was easier to finish and run safely.
It would be a one-off job that you'd run just once, to handle most people with blank mission data. It wouldn't need an id for each unique mission yet, because you know you are picking all users that have completely blank missions and are running it once.

Pseudo code would look something like:
for all users, once:
-check if ALL missions are blank. if not, skip to next user
-fill mission data:
--if rajas or title: cop=end
--if moonshade or title: wotg=end
--etc

After that runs, you'd have solved most of the problem as lots of users will now have ranking.

Then you can focus on the harder task of the hourly update with checking of unique ids per mission etc, which is important but less urgent and more like the last 10% that takes 90% of the work.
 Asura.Chiaia
VIP
Offline
Server: Asura
Game: FFXI
user: Demmis
Posts: 1652
By Asura.Chiaia 2020-11-30 05:16:22
Link | Quote | Reply
 
Rooks said: »
What I have to do now, is a bunch of those checks. No particular one is hard, but they're all a little diffferent: I have to call up people's titles, check their whole list of items to see if they have a Rajas Ring, etc. I don't want to potentially do all of this math every time someone hits //gw flush or zones, so hourly with the achievement processor is the correct answer.
Any reason you're still intending to use titles/(ear)rings and not the mission packet? That way it can still work like it use to when SE LS beta gave that data and show current mission. I gave you a complete mapping for missions. Well it is missing "The Voracious Resurgance".but that's super new and none complete.

Just curious as why you see one better then the other especially when one can only handle pretty much completed or not.
[+]
 Asura.Arico
Offline
Server: Asura
Game: FFXI
user: Tename
Posts: 535
By Asura.Arico 2020-11-30 05:36:38
Link | Quote | Reply
 
Asura.Chiaia said: »
Just curious as why you see one better then the other especially when one can only handle pretty much completed or not.

Probably because he can use titles without touching the guildwork plugin.
[+]
 Leviathan.Isiolia
Offline
Server: Leviathan
Game: FFXI
user: Isiolia
Posts: 458
By Leviathan.Isiolia 2020-11-30 06:30:21
Link | Quote | Reply
 
Are Records of Eminence recorded in a way that’d readily allow for that? Obviously it’s not quite the same as automatically figuring missions out and all, but it shouldn’t be a big deal for people that care to do them. Missions they’d likely make a point to for the CP bonus anyway, but there’s stuff like WS unlocks or content progression that have ROEs as well.
 Asura.Arico
Offline
Server: Asura
Game: FFXI
user: Tename
Posts: 535
By Asura.Arico 2020-11-30 06:57:27
Link | Quote | Reply
 
Leviathan.Isiolia said: »
Are Records of Eminence recorded in a way that’d readily allow for that? Obviously it’s not quite the same as automatically figuring missions out and all, but it shouldn’t be a big deal for people that care to do them. Missions they’d likely make a point to for the CP bonus anyway, but there’s stuff like WS unlocks or content progression that have ROEs as well.

Your entire mission/quest list is sent to you every time you zone or complete/accept a mission/quest. It would be better than RoE since there are some people too lazy to do the RoEs. Also the RoE IDs haven't been fully mapped(if someone did please send them to me <3).
 Asura.Chiaia
VIP
Offline
Server: Asura
Game: FFXI
user: Demmis
Posts: 1652
By Asura.Chiaia 2020-11-30 07:05:49
Link | Quote | Reply
 
RoE are bit packed and I don't know anyone that has went through and figured them all out. Still a worse idea then using the Mission/Quest packet for those 2 things though. If he wanted to start recording RoE progress (added feature) then he or someone else would have to map it all out.
Administrator
Offline
Posts: 6495
By Rooks 2020-11-30 07:20:15
Link | Quote | Reply
 
Asura.Chiaia said: »
Any reason you're still intending to use titles/(ear)rings and not the mission packet? That way it can still work like it use to when SE LS beta gave that data and show current mission. I gave you a complete mapping for missions. Well it is missing "The Voracious Resurgance".but that's super new and none complete.

Conservation of an extremely limited resource: my time.

The ACH parser had to be overhauled/titles had to be fixed anyway, the current parser skips and loses people for all sorts of reasons. Modifying the GW plugin and both APIs AND the ACH parser to use the mission info is the eventual goal, but this gets me to a workable solution faster.

So, you're right about the destination, but it's a step or two away.
[+]
 Cpu
Offline
Posts: 248
By Cpu 2020-11-30 08:41:43
Link | Quote | Reply
 
Rooks said: »
Asura.Chiaia said: »
Any reason you're still intending to use titles/(ear)rings and not the mission packet? That way it can still work like it use to when SE LS beta gave that data and show current mission. I gave you a complete mapping for missions. Well it is missing "The Voracious Resurgance".but that's super new and none complete.

Conservation of an extremely limited resource: my time.

The ACH parser had to be overhauled/titles had to be fixed anyway, the current parser skips and loses people for all sorts of reasons. Modifying the GW plugin and both APIs AND the ACH parser to use the mission info is the eventual goal, but this gets me to a workable solution faster.

So, you're right about the destination, but it's a step or two away.
Put it on github and we'll do it for you while also preserving site integrity by making you a required PR reviewer or the only user with merge access?
Administrator
Offline
Posts: 6495
By Rooks 2020-11-30 08:51:30
Link | Quote | Reply
 
Cpu said: »
Put it on github and we'll do it for you while also preserving site integrity by making you a required PR reviewer or the only user with merge access?
That's not my decision to make.

And it's on Github already, as a private repo - I send PRs to Scragg, who approves and deploys them. I don't have production access beyond anything I build a tool for in the site itself.
 Cpu
Offline
Posts: 248
By Cpu 2020-11-30 08:56:50
Link | Quote | Reply
 
Rooks said: »
Cpu said: »
Put it on github and we'll do it for you while also preserving site integrity by making you a required PR reviewer or the only user with merge access?
That's not my decision to make.

And it's on Github already, as a private repo - I send PRs to Scragg, who approves and deploys them. I don't have production access beyond anything I build a tool for in the site itself.
Being a private repo has one big advantage, in that you avoid potential exploiters having access to the codebase, but that seems like a pretty minor upside when you also have the same opportunity for security improvements via public codebase reviews. But if it's not your decision to make then oh well.
 Bismarck.Xurion
Offline
Server: Bismarck
Game: FFXI
user: Xurion
Posts: 693
By Bismarck.Xurion 2020-11-30 09:05:11
Link | Quote | Reply
 
Asura.Chiaia said: »
RoE are bit packed and I don't know anyone that has went through and figured them all out. Still a worse idea then using the Mission/Quest packet for those 2 things though. If he wanted to start recording RoE progress (added feature) then he or someone else would have to map it all out.
Have quests and/or missions been mapped?
Offline
Posts: 1584
By Felgarr 2020-11-30 09:39:59
Link | Quote | Reply
 
Good luck you guys. I've been barking up this tree for years and never actually got a reply from Scragg.
 Asura.Arico
Offline
Server: Asura
Game: FFXI
user: Tename
Posts: 535
By Asura.Arico 2020-11-30 14:06:17
Link | Quote | Reply
 
Bismarck.Xurion said: »
Have quests and/or missions been mapped?

By a couple people. None of them are exactly the same, and from what I can see, most SoA quests are wrong.
 Asura.Sechs
Offline
Server: Asura
Game: FFXI
user: Akumasama
Posts: 9876
By Asura.Sechs 2020-11-30 18:46:55
Link | Quote | Reply
 
In the long run, RoE mapping sounds like the best option.
Allows a tool like guildwork (or anything else) to parse how deep you're in a specific mission and a plethora of other things that could be useful for FFXIAH as well.

I get this would require a lot of work in parsing packets and find where RoE data is stored and so on.
My question is: can anybody of us help with that? If someone were to organize work and divide it in steps I'd be more than willing to contribute.
 Asura.Chiaia
VIP
Offline
Server: Asura
Game: FFXI
user: Demmis
Posts: 1652
By Asura.Chiaia 2020-12-01 03:15:24
Link | Quote | Reply
 
Asura.Sechs said: »
In the long run, RoE mapping sounds like the best option.
For mission data it is definitely not the best method and even other things it can show you. It would be good imo just to show what/how many RoEs you've completed.

Asura.Sechs said: »
My question is: can anybody of us help with that? If someone were to organize work and divide it in steps I'd be more than willing to contribute.
Anyone can really take up this project since RoE is an incoming packet just jump on a private server with the latest client and start injecting the RoE packet which can be found in fields.lua with altered info and check out the RoE menu in game. I'm leaving this super vague because if you don't already have a grasp of what I'm talking about then you probably shouldn't be someone trying to help with it.
 Asura.Chiaia
VIP
Offline
Server: Asura
Game: FFXI
user: Demmis
Posts: 1652
By Asura.Chiaia 2020-12-01 03:25:51
Link | Quote | Reply
 
Bismarck.Xurion said: »
Asura.Chiaia said: »
RoE are bit packed and I don't know anyone that has went through and figured them all out. Still a worse idea then using the Mission/Quest packet for those 2 things though. If he wanted to start recording RoE progress (added feature) then he or someone else would have to map it all out.
Have quests and/or missions been mapped?
Not that I'm aware of for RoE related stuff(packet). I meant the mission/quest packet.

I know missions are set for the mission packet, at least the version I have since I helped on it but I think only 3 of us have it. The person I helped with it, myself, and Rooks when I sent it his way though I may have shared it with all of the Windower team at the same time though.

Quest I know 1 or 2 have data on but it also didn't match up sometimes.
 Asura.Arico
Offline
Server: Asura
Game: FFXI
user: Tename
Posts: 535
By Asura.Arico 2020-12-01 11:39:32
Link | Quote | Reply
 
Asura.Chiaia said: »
Anyone can really take up this project since RoE is an incoming packet just jump on a private server with the latest client and start injecting the RoE packet which can be found in fields.lua with altered info and check out the RoE menu in game. I'm leaving this super vague because if you don't already have a grasp of what I'm talking about then you probably shouldn't be someone trying to help with it.

If anyone wants to do this I have a script that will set every RoE and take a screenshot of the roe menu. Takes about 45minutes to run and however long it takes you to go through the images. I'd recommend not doing it on live, and I know nothing about roes on private servers.
necroskull Necro Bump Detected! [266 days between previous and next post]
 Siren.Codegen
Offline
Server: Siren
Game: FFXI
user: Genome
Posts: 180
By Siren.Codegen 2021-08-24 03:38:32
Link | Quote | Reply
 
Any chance of this below from 9 months ago still happening at some point? It still seems just as useful.
Siren.Codegen said: »
Rooks said: »
Siren.Codegen said: »
no chance of overloading resources as it's at most a once per char computation
This is why I say:

Rooks said: »
specifically only calculating for achievements you don't currently have, e.g., once I flag you as having completed CoP, I never have to check for a Rajas Ring again

Right, and I understand you already coded part of your approach, I was just suggesting an even easier intermediate option if that was easier to finish and run safely.
It would be a one-off job that you'd run just once, to handle most people with blank mission data. It wouldn't need an id for each unique mission yet, because you know you are picking all users that have completely blank missions and are running it once.

Pseudo code would look something like:
for all users, once:
-check if ALL missions are blank. if not, skip to next user
-fill mission data:
--if rajas or title: cop=end
--if moonshade or title: wotg=end
--etc

After that runs, you'd have solved most of the problem as lots of users will now have ranking.

Then you can focus on the harder task of the hourly update with checking of unique ids per mission etc, which is important but less urgent and more like the last 10% that takes 90% of the work.
Also side note, the guildwork flush seems to have stopped working sometimes on August 10, the plugin still responds to commands like help, but flush does nothing, not even the message "flushing" shows up anymore, an no data has been updated at least on my account, not sure if it's just me or everyone else.
First Page 2
Log in to post.