It's been a good month since the character tracking stuff has gone into XIVDB and for the most part it went pretty well!
The timing for releasing all the new site stuff was the same day FFXIV 3.4 landed, so huge traffic spikes caused the site to burn up a little :D - after that it was mostly smooth! Traffic quadrupled during this patch which is higher than normal (usually it only doubles), which was very nice! Lots of new comments!
Since the implementation, there have been some minor tweaks:
- Gearsets now detect job roles
- Layout of gearsets tweaked slightly
- Ability to delete gearsets
- Fixed issues for timeline with a new characters initial events
- Ability to manually update yourself!
There are a few issues I still have to solve:
- Mount tracker lists mounts you can't get anymore, eg: Gooboo from 1.0 and Legacy Mount, need to separate these
- Commendation achievements give 0 points, so to avoid division by 0, you have 0% completion! :D
- Some achievements are "one in a series", example City mission achievements. Right now you can only get 1, maybe in the future all 3. I was going to hide the others if you had one, but the further issue is if you played Uldah in 1.0 but switched to Limsa in 2.0 you can get 2/3 achievements and I don't want to break that. So got to think about it
- Estimated time until full level 60. This was in the old site but it was very inaccurate, this is because I can only predict your leveling curve based on what you're currently leveling. If you have 10 classes not max'd out, but only leveling 3, I would have to "guess" the exp of your other 7, depending on if these are gathering, crafting or battle classes, they all level at very different speeds and so it becomes complicated, soon!(tm)
When I originally ran my XIVPads site I had XIVSync on 1 single server (a pretty beefy one). It mostly worked fine but over time it would become a bit sluggish as there were a lot of actions trying to fight for resources: Parse lodestone, update database, provide an api, push to XIVDB, manage data, etc. Along with that it also contained all the parsing data, which was then pushed onto XIVDB, which means duplication of data! (Which is near 200GB worth!)
After fixing a bunch of stuff on the site and being a little fed up of programming, I decided to tackle this Server structure. What I wanted to do was split the sync services up into multiple cloud servers for each different task, this means a single server would be dedicated to a specific role. Fortunately when I wrote my code it would be pretty easy to do this, most of the work was in infrastructure setup, fail overs and stuff like that.
So now! XIVSync is multi-server!!
The benefit to this; it costs less and is much more easier to scale. If i want to parse more character then I simply add another server to do it, each character parser can do around 350-400 characters a minute (depends on latency and lodestone traffic), right now I run 3 of those. When the next patch drops or even the expansion, I can speed up character updating and then as things cool down I can easily reduce it as well :)
Here is how the setup looks:
You can view the status of XIVSync either:
- Going to https://xivsync.com/
- Looking on the right side of the home page (near the bottom) on XIVDB.com
Moving onto something slightly different
So after burning out on programming, and then burning out on server stuff I wanted to try out something different. It has always bugged me the lack of NPC and Enemy data on the site. The main issue is: Enemy data is heavily server side and NPC data is all over the place and not cleanly in the dats.
One thing I wanted to do was heavily populate the maps and get some other neat info like enemy levels, hp, mp, etc.
To do this, it means reading the games memory! So over the past couple days I patched up a memory module from FFXIV-APP and built a self contained application that will record any Enemy or NPC data around me and submit it to XIVDB!
This should greatly improve the data. I likely won't release this to public as it would heavily hammer XIVDB servers, and since Enemy/NPC positions don't change often then scanning only need to be done once. So just a few helpers will run it and map out a zone. (3.0 flying zones are sooo easy!)
Another task involved this is concatenating a whole bunch of NPC data. SE like to duplicate NPCs for every map/quest/mission they're in, this means some npcs (cough Alphinaud cough) have hundreds of NPC entries. I'd like there to be just 1, that provides information on all whereabouts. So that will be the task after :)
Here is a little look at the application
I am really excited to get this pushing to XIVDB and have real positions.
In closing, lots to be done
The next couple weeks I am going to be focusing soley on game data, fix up a lot of that and get more into the site. I haven't touched it in a while and I don't like how some stuff is really empty. I would also like to link more data up (minions to items, cards to items, quest interactions, etc)
It's my Sunday off, time to play some FFXIV :D
Enjoy the rest of your weekend!