SeAT - forum: \Seat\Web\Jobs\UpdateCharacterFilters failure - Page 1

@user_154092528386310154 a few days ago you reported that \Seat\Web\Jobs\UpdateCharacterFilters fails for you and squads don't update anymore. Does the issue still persist for you? I'd like to investigate this. In short: \Seat\Web\Jobs\UpdateCharacterFilters is responsible for updating squads, so these two issues are related.

You also mentioned out-of-memory issues. Back then, I asked for a complete stack trace of the OOM error, but I haven't noticed one yet. During testing, we did run into a OOM error once, so it might be related. I really need a full stack trace to confirm that.

also, do you have huge

Can I also ask if you use many squads with huge squad filters?

ok thanks. I'm just trying to rule out things

[2024-09-22 19:38:42] local.ERROR: Allowed memory size of 536870912 bytes exhausted (tried to allocate 2452656 bytes) {"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Allowed memory size of 536870912 bytes exhausted (tried to allocate 2452656 bytes) at /var/www/seat/vendor/laravel/framework/src/Illuminate/Database/Connection.php:428)

is that everything? It doesn't look complete

Yea

heres -A 10 -B 10

#45 /var/www/seat/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\\Container\\Container->call(Array)
#46 /var/www/seat/vendor/symfony/console/Command/Command.php(326): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#47 /var/www/seat/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#48 /var/www/seat/vendor/symfony/console/Application.php(1096): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#49 /var/www/seat/vendor/symfony/console/Application.php(324): Symfony\\Component\\Console\\Application->doRunCommand(Object(Laravel\\Horizon\\Console\\WorkCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#50 /var/www/seat/vendor/symfony/console/Application.php(175): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#51 /var/www/seat/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#52 /var/www/seat/artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#53 {main}
"} 
[2024-09-22 19:38:42] local.ERROR: Allowed memory size of 536870912 bytes exhausted (tried to allocate 2452656 bytes) {"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Allowed memory size of 536870912 bytes exhausted (tried to allocate 2452656 bytes) at /var/www/seat/vendor/laravel/framework/src/Illuminate/Database/Connection.php:428)
[stacktrace]
#0 {main}
"} 
[2024-09-22 19:41:30] local.ERROR: The datasource tranquility is temporarily unavailable {"exception":"[object] (Seat\\Eveapi\\Exception\\UnavailableEveServersException(code: 503): The datasource tranquility is temporarily unavailable at /var/www/seat/vendor/eveseat/eveapi/src/Jobs/EsiBase.php:461)
[stacktrace]
#0 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/EsiBase.php(323): Seat\\Eveapi\\Jobs\\EsiBase->handleEsiFailedCall(Object(Seat\\Eseye\\Exceptions\\RequestFailedException))
#1 /var/www/seat/vendor/eveseat/eveapi/src/Jobs/Character/Notifications.php(74): Seat\\Eveapi\\Jobs\\EsiBase->retrieve(Array)
#2 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Seat\\Eveapi\\Jobs\\Character\\Notifications->handle()
#3 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#4 /var/www/seat/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))

hmm. weird that it doesn't give a full stack trace. @user_301981661761896449 , do you have any idea why that could be?

I also went through and re-saved every squad and filter, doing lil edits on the way just to see if I could get one to error

no errors showed up

thanks, I think for now I have everything I need to know

another question: How often do you get the OOM issues? A few times an hour, every minute, once a day?

looks like about 3 times an hour

That makes me 95% sure it is the squad updates job

to remove the last 5%, I actually added a new debug tool to verify. It is merged, but not yet released: https://github.com/eveseat/eveapi/pull/414

installed it, will see what it does

In that case, can you run php artisan esi:job:dispatch --sync "Seat\Web\Jobs\UpdateCharacterFilters" and report the result?

php artisan esi:job:dispatch --sync "Seat\Web\Jobs\UpdateCharacterFilters" The job is not part of Esi stack. In Connection.php line 430: Allowed memory size of 536870912 bytes exhausted (tried to allocate 2451456 bytes)

ok thanks. So that confirms that the squad updates are the problem. I'll write a fix shortly

Would you be willing to, for a test, 1) double the memory limit, 2) run the command from above to see if it works 3) if it does, report me how many times you had to increase the memory limit 4) if it doesn't, repeat step 1 (unless it does get silly/your machine can't support it) To get an estimate by how much we need to reduce memory usage? We have an easy knob for decreasing memory usage, but we need to figure out how much I need to turn it down

Sure

Default is 128

I'll do that tonight though

passes at 2048M

errors at 1500

https://github.com/eveseat/web/pull/689