Bad lag?

Anette

Sorry, couldn't really ask ingame and I don't know if this is the proper place to ask, but I've been having some really gnarly lag playing over the weekend that seems to have just now culminated in like, maybe every 5th command getting through (and I'm sure my spamming doesn't help, so I made my offerings and qq'd).

Am I alone in having that?  Ping to Imperian is nominal (15ms) and I don't seem to have any packet loss.  Lag might not even be the term if its something janky with the game but it seems the closest.  Originally given the commands taking so much to go through I thought maybe my system was acting up; but the site's play interface does the same thing.  And it logs in and qq's fine.  Something with the command parser maybe?  Iunno.

Either way, is this just me, or...?

Anette

So the commands stuff is a concussion, checked that on a hunch.  Trying to cure that out of resto and having to wait ~30s for commands to go through when only one in so many is going to do so anyways b/c of the concussion is an exercise in frustration, though.

Jules

Anette, the lag got bad maybe, a month or more so ago now? Not sure why. But it is not as frequent as you seem to be experiencing.  I recently (finally) installed a wifi range extender, and I am pleased with the results.  There is a lot of channel overlap/interference/chatter in my neighborhood, for example, and while switching channels would often fix the problem for a while, having a local signal boost (that I hopefully adjusted properly not to overwhelm my neighbors in an escalating Mutually Assured Denied Interwebs war) seems to have helped a lot. 

Anette

Got it bad now again, and literally all I'm doing is sitting at Conquest's altar talking with people on ring.  >30s for stuff to go through. 

Took me like two minutes just to get back on Impy just to tell em that I lagged out.

[edit]: Quick network test, it definitely is not ping:

28 packets transmitted, 28 received, 0% packet loss, time 27825ms

rtt min/avg/max/mdev = 13.900/14.057/14.304/0.138 ms

13ms ping is divine.  So what gives?

[edit #2]: Letting it go on for longer, there's some real spikes of significant packet loss, probably related.

Tried it from both home (Ottawa, Canada) and my server (Kansas City, USA) and there's the same loss.  :x  That'll do 'er.

[edit #3] Looks like box (probably edge router) at pr1.chi2.servernap.net [206.223.119.108] is the problem spot, for any admins reading.

Septus

So, this is definitely a thing. Demonic and am were squaring off for a proper slugfest and the lag was so bad that at least for me, I could enter a command on entering and it'd happen upwards of 45 seconds later. My ping doesn't fluctuate at all, and by reports it was hitting just about everyone. I'm assuming there is something inefficient somewhere (only happened with the large groups of us in the same room, but made things pretty much unplayable).

@Garryn
@Jeremy
@Dec

Salik

We were all noticing this, and talking about it. It wasn't lagging until you guys came into the area. We're fairly sure it's related to relic code somehow.

Linslet

That slugfest felt like we were fighting in old retardation vibes.  The nostalgia factor made me happy.

Anette

Septus said:

So, this is definitely a thing. Demonic and am were squaring off for a proper slugfest and the lag was so bad that at least for me, I could enter a command on entering and it'd happen upwards of 45 seconds later. My ping doesn't fluctuate at all, and by reports it was hitting just about everyone. I'm assuming there is something inefficient somewhere (only happened with the large groups of us in the same room, but made things pretty much unplayable).

@Garryn
@Jeremy
@Dec

Seems to be more the command processor than actual lag when that happens.  I've died a few times because the game just decided to stop accepting my inputs, despite the connection being sound.

In fact as I write this my blessings for defences have hung up input handily, and the connection itself is sound. 35~50ms ping, no packet loss.

Speculation: This seems to happen on the world ticks.  Might be something in the game logic there is hanging up badly but only occasionally.

Garryn

It was the message cleanup, should be resolved for now.

Anette

Garryn said:

It was the message cleanup, should be resolved for now.

Here's hoping.  Thanks for looking into this!

Ohm

Still seem to be experiencing bad lag spikes .. not sure if this is the case with everyone though

Ultrix

Still having pretty significant lag spikes going on.  In particular, our little shrine skirmish last night in Caanae had lots of lag whereby input sent to the server would show up a good 30 seconds later.  I agree with @Salik that it likely has to do with shrines/relic code -- the "banish" relic seems to cause a lot of lag when it is activated, for example.

Anette

Something checked on world-tick seems off.  I get this input-lag even when there's like six people playing this morning.  I'll still be server messages fine but it takes a full 30s to sometimes even a minute to register my commands.  I'm glad that there is no death penalty, because Ive been eating a frustrating amount of deaths I really shouldnt be as a result.

Ohm

Anette said:

Something checked on world-tick seems off.  I get this input-lag even when there's like six people playing this morning.  I'll still be server messages fine but it takes a full 30s to sometimes even a minute to register my commands.  I'm glad that there is no death penalty, because Ive been eating a frustrating amount of deaths I really shouldnt be as a result.

This - seems to occur every half past the hour (EST)

Anette

If I had to take a guess based on the fact that I just got disconnected during one of those moments that it's trying to do a copyover, lags loading it, and in mt recent case, lost the connection.

Anette

PS the lag is like, really super bad today, blimey.  Almost every 15min like clockwork.

Linslet

Ditto on the "eating deaths I shouldn't" comment.  The absurd input lag just caused me to be unable to exit a room with more demons than anyone would be comfortable with.

Anette

Linslet said:

Ditto on the "eating deaths I shouldn't" comment.  The absurd input lag just caused me to be unable to exit a room with more demons than anyone would be comfortable with.

It's kind of super frustrating when the only reason is the input lag

Dicene

We're pretty solidly in "this lag is so bad that teamfights are not even worth participating in at the moment because i'm hitting ward with every single combo and sometimes my input takes so long to go through that I've been targeted and killed before getting a chance to respond" territory. Plzfix

Linslet

Alright, this lag is to the point where fights are unplayable.  Half the demonic team just had their offenses negated by curseward of all things because their input wouldn't go through until garrynbot had already put curseward back up.

It's interfering with bashing.  It's interfering with any form of headway in fights.  I'm hesitant to allocate any more resources to either until I know the payoff won't be at the mercy of lag.

Anette

Tis why I havent been about ingame too much.  Not much point if Im just going to die to input lag.

Dec

Lag is notoriously hard to track down because there is a whole lot that could cause it and very little evidence left after the fact.  Timestamped logs showing the lag and what was going on before and after (especially if its in a large combat) would be a start.  Please either post (preferrably links) or send me any logs that might show what was happening when the lag started.  Or if you think you see something causing it please let me know your theories.  It's good to know where the fights happen, what is involved in the fights (e.g. shrines) and any other useful information.  I have the imperian.com email address dec if you want to email something.

Septus

Its hard to pin anything down because its pretty much the instant a fight starts. For instance, I walked into a fight the other day, hit converge/arc ciguatoxin, and hit someone's deliverance with the arc 1.5 mins later (I saw them start it about 45 seconds after hitting the command).

The fights that I've personally witnessed it in happened either in dp or caanae. No room based relics were in play in either case (there were shrines in the rooms, but they only ever had aegis. One fight might have happened in a room with a banish relic in the shrine (depends if we moved at some point during the initial lag), but it was on cooldown for certain.

I personally don't witness any lag bashing in heavily shrined areas (Necropolis, dp), but a fair few others say that they do. I'm aware that's pretty anecdotal though.

Ohm

Kicks in every 33rd minute of the hour like clockwork (EST)

Anette

@Dec - 

Ohm said:

Kicks in every 33rd minute of the hour like clockwork (EST)

This.  To be specific, the lag I experience is a bit odd, and intermittent in behaviour.

I get one of three flavours of it:

1] The game drops my connection entirely after a short interval ("Connection reset by peer")

2] The game seems to freeze entirely for 10~30 second including both input and output (this is the rarer of the three but I've still had happen a few times)

3] The game seems to take a long time to process my inputs (say for instance dsl horde), but output continues, so it takes me twice as long to hit the orcs back but they hit me just fine.  Yeah, this ones seen me die a few times.

There is no real pattern to WHAT I'm doing when it occurs, but I think @Ohm is on the right track suggesting it's timed, because I've observed that as well.

[edit]: My logs are verbose so I'm not sure it's the greatest idea to post them on the forums, but I have no problem with tossing it at support@imperian.com or an alternative email if you prefer next time I observe it happening if it would at all be helpful.

[edit 2]: Just to clarify, my suspicion is that it's something happening on the game heartbeat checks.  I suspect group combat probably exacerbates it, but I've also gotten this lag at times when there's literally 6 people on the server.  I'd rather say, it's probably more something related to combat abilities.  

Entirely anecdotal, but I've noticed the input lag seems worse when I'm bleeding a lot.  This could be something related to the checks for clotting ticks, or it could just be because I get spammy with f4 which is my clot button making it harder for commands to get through, it's hard to say.

Next time I notice it happening I'll try to take some notes as it were.  Is there anything in particular that would be helpful to you for me to look out for?

Khizan

I don't have the fight log, but this is the deathsight from a big laggy fight in the Byley Plains last night. I don't think anybody has shrines there but I did not see for certain.

2016/03/11 02:14:01 - Abigail has been slain by Ultrix.

2016/03/11 02:14:07 - Abigail has been slain by Razahk.

2016/03/11 02:18:40 - Dexter has been slain by a horned beast.

2016/03/11 02:21:24 - Razahk has been slain by Linslet.

2016/03/11 02:22:37 - Arcann has been slain by Pellerin.

2016/03/11 02:22:54 - Linslet has been slain by Ultrix.

2016/03/11 02:28:28 - Cyr has been slain by Linslet.

2016/03/11 02:28:33 - Arcann has been slain by Ultrix.

2016/03/11 02:28:48 - Kabaal has been slain by Razahk.

2016/03/11 02:29:13 - Cyr has been slain by Linslet.

2016/03/11 02:29:45 - Linslet has been slain by Ultrix.

2016/03/11 02:29:57 - Pellerin has been slain by Baasche.

2016/03/11 02:29:58 - Linslet has been slain by Razahk.

2016/03/11 02:30:20 - Baasche has been slain by Khizan.

2016/03/11 02:31:44 - Sevhn has been slain by Salik.

Anette

Froze for a moment then booted me just now

2016-03-11 19h24:17 | SEND | perform ritual protection favour
2016-03-11 19h24:18 | RECV | You focus on the strength of your belief, and soon enough feel more safe as your faith drives you onwards.
2016-03-11 19h24:21 | RECV | 
2016-03-11 19h24:21 | RECV | You have recovered balance.
2016-03-11 19h24:21 | SEND | 
2016-03-11 19h24:26 | SEND | perform ritual fortune favour
2016-03-11 19h25:19 | DISC | 

Happened at my house, licking my wounds after dying to Rafe (did someone up his damage? He was doing a lot more than I usually deal with) Can't give a vnum, it's not mapped. Room has protection addon. QW when I got back on:

-------------------------------------------------------------------------------
Abigail, Alexys, Anette, Arakis, Ario, Bethanee, Caelbrook, Calisyn, Camaris, Cyr, Dagr, Durlar,
    Essray, Galador, Gurn, Iniar, Jebadiah, Kabaal, Kalima, Kasdeya, Katinel, Kezak, Khizan,
    Laerten, Lasine, Lucian, Lyaewyn, Ohm, Oystir, Pingas, Rai, Razahk, Robynn, Sela, Sevhn,
    Shaheen, Tikal, Torseno, Vasharr, Veratyr, and Zooeyah.
-------------------------------------------------------------------------------
There are 44 players in this world.

[edit] FWIW, room is ANETTE_HOUSE_1_1_3 - but I don't think it's relevant.

Andarius

Can confirm at xx:33 the entire game seems to lag for 10-15 seconds. Was doing this while bashing and can tell it's server side because I didn't end up dead, they only took the one action in that time span.

Anette

Yeah irrespective of other lag instances that xx:33 one seems pretty consistent, albeit thankfully short and since it's server-side, not something that gets me killed.

Khizan

I never noticed it before, but confirming big lag spike at 16:33 CST.

Dec

I've been going through the code to try to improve the delays I can come across.  Not sure if anything has actually helped so far. 

I've seen the x:33 lag, although it's not consistent for me.  There's nothing obvious running at 33 minutes past the hour, but there's still plenty of places to look.

Note that I cannot deal with network lag.  This would be things like disconnects or where the game keeps going, but your inputs don't seem to register.  I am trying to deal with the situations where the game seems to just halt (both in and out) for a few seconds before churning on.  I know network lag is annoying too, but I can only deal with problems with the game itself.