Page MenuHomeMiraheze

Editing and contributions broken on FAMEPedia
Closed, ResolvedPublic

Description

Shows an error.

[db1a142711000db3e921bfd1] 2021-10-17 09:37:21: Fatal exception of type "Error"

Event Timeline

RhinosF1 triaged this task as High priority.Oct 17 2021, 09:38
RhinosF1 created this task.
Herald added subscribers: Unknown Object (User), Bongo-Cat, Unknown Object (User) and 2 others. · View Herald TranscriptOct 17 2021, 09:38
Call to undefined method HistoryBlobStub::uncompress()
from /srv/mediawiki/w/includes/historyblob/HistoryBlobStub.php(129)
#0 /srv/mediawiki/w/includes/Storage/SqlBlobStore.php(608): HistoryBlobStub->getText()
#1 /srv/mediawiki/w/includes/Storage/SqlBlobStore.php(520): MediaWiki\Storage\SqlBlobStore->decompressData(string, array)
#2 /srv/mediawiki/w/includes/Storage/SqlBlobStore.php(430): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#3 /srv/mediawiki/w/includes/Storage/SqlBlobStore.php(286): MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, integer)
#4 /srv/mediawiki/w/includes/libs/objectcache/wancache/WANObjectCache.php(1707): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL, array)
#5 /srv/mediawiki/w/includes/libs/objectcache/wancache/WANObjectCache.php(1539): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#6 /srv/mediawiki/w/includes/Storage/SqlBlobStore.php(291): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#7 /srv/mediawiki/w/includes/Revision/RevisionStore.php(1191): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#8 /srv/mediawiki/w/includes/Revision/RevisionStore.php(1463): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#9 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#10 /srv/mediawiki/w/includes/Revision/SlotRecord.php(324): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#11 /srv/mediawiki/w/includes/Revision/RevisionRecord.php(164): MediaWiki\Revision\SlotRecord->getContent()
#12 /srv/mediawiki/w/includes/parser/Parser.php(3697): MediaWiki\Revision\RevisionRecord->getContent(string)
#13 /srv/mediawiki/w/includes/parser/Parser.php(3547): Parser->statelessFetchTemplate(Title, Parser)
#14 /srv/mediawiki/w/extensions/Scribunto/includes/common/ScribuntoEngineBase.php(167): Parser->fetchTemplateAndTitle(Title)
#15 /srv/mediawiki/w/extensions/Scribunto/includes/engines/LuaCommon/LuaEngine.php(571): ScribuntoEngineBase->fetchModuleFromParser(Title)
#16 /srv/mediawiki/w/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxCallback.php(26): Scribunto_LuaEngine->loadPackage(string)
#17 [internal function]: Scribunto_LuaSandboxCallback->__call(string, array)
#18 /srv/mediawiki/w/extensions/Scribunto/includes/engines/LuaSandbox/LuaSandboxInterpreter.php(113): LuaSandboxFunction->call(LuaSandboxFunction)
#19 /srv/mediawiki/w/extensions/Scribunto/includes/engines/LuaCommon/LuaEngine.php(295): Scribunto_LuaSandboxInterpreter->callFunction(LuaSandboxFunction, LuaSandboxFunction)
#20 /srv/mediawiki/w/extensions/Scribunto/includes/engines/LuaCommon/LuaModule.php(68): Scribunto_LuaEngine->executeFunctionChunk(LuaSandboxFunction, PPTemplateFrame_Hash)
#21 /srv/mediawiki/w/extensions/Scribunto/includes/common/Hooks.php(140): Scribunto_LuaModule->invoke(string, PPTemplateFrame_Hash)
#22 /srv/mediawiki/w/includes/parser/Parser.php(3356): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#23 /srv/mediawiki/w/includes/parser/Parser.php(3041): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#24 /srv/mediawiki/w/includes/parser/PPFrame_Hash.php(273): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#25 /srv/mediawiki/w/includes/parser/Parser.php(3230): PPFrame_Hash->expand(PPNode_Hash_Tree)
#26 /srv/mediawiki/w/includes/parser/PPFrame_Hash.php(273): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#27 /srv/mediawiki/w/includes/parser/Parser.php(3230): PPFrame_Hash->expand(PPNode_Hash_Tree)
#28 /srv/mediawiki/w/includes/parser/PPFrame_Hash.php(273): Parser->braceSubstitution(array, PPFrame_Hash)
#29 /srv/mediawiki/w/includes/parser/Parser.php(2879): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#30 /srv/mediawiki/w/includes/parser/Parser.php(1549): Parser->replaceVariables(string)
#31 /srv/mediawiki/w/includes/parser/Parser.php(639): Parser->internalParse(string)
#32 /srv/mediawiki/w/includes/OutputPage.php(2115): Parser->parse(string, Title, ParserOptions, boolean, boolean, NULL)
#33 /srv/mediawiki/w/includes/OutputPage.php(1864): OutputPage->parseInternal(string, Title, boolean, boolean)
#34 /srv/mediawiki/w/includes/OutputPage.php(1795): OutputPage->addWikiTextTitleInternal(string, Title, boolean, boolean)
#35 /srv/mediawiki/w/includes/OutputPage.php(4128): OutputPage->addWikiTextAsInterface(string)
#36 /srv/mediawiki/w/includes/specials/SpecialContributions.php(368): OutputPage->wrapWikiMsg(string, array)
#37 /srv/mediawiki/w/includes/specialpage/SpecialPage.php(646): SpecialContributions->execute(string)
#38 /srv/mediawiki/w/includes/specialpage/SpecialPageFactory.php(1386): SpecialPage->run(string)
#39 /srv/mediawiki/w/includes/MediaWiki.php(309): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#40 /srv/mediawiki/w/includes/MediaWiki.php(913): MediaWiki->performRequest()
#41 /srv/mediawiki/w/includes/MediaWiki.php(546): MediaWiki->main()
#42 /srv/mediawiki/w/index.php(53): MediaWiki->run()
#43 /srv/mediawiki/w/index.php(46): wfIndexMain()
MariaDB [famepediawiki]> SELECT * FROM text WHERE old_flags = 'object,utf-8' AND old_text NOT LIKE '%ConcatenatedGzipHistoryBlob%' LIMIT 3;
+--------+---------------------------------------------------------------------------------------------------------------------------------------+--------------+
| old_id | old_text                                                                                                                              | old_flags    |
+--------+---------------------------------------------------------------------------------------------------------------------------------------+--------------+
|     50 | O:15:"HistoryBlobStub":3:{s:9:" * mOldId";s:5:"24648";s:8:" * mHash";s:32:"4fd6bef120de43768abbbd24da40051f";s:7:" * mRef";s:2:"50";} | object,utf-8 |
|     72 | O:15:"HistoryBlobStub":3:{s:9:" * mOldId";s:5:"24167";s:8:" * mHash";s:32:"17cbe2208cfbd26d5a5a052188c9b198";s:7:" * mRef";s:2:"72";} | object,utf-8 |
|     99 | O:15:"HistoryBlobStub":3:{s:9:" * mOldId";s:5:"24486";s:8:" * mHash";s:32:"72ca804c7ef14b83ff649bc3e013a9c5";s:7:" * mRef";s:2:"99";} | object,utf-8 |
+--------+---------------------------------------------------------------------------------------------------------------------------------------+--------------+

If I had to guess, these 3 revision entries would be the problems.

Ignore the above - that is expected actually.

Unknown Object (User) renamed this task from Special:Contributions on famepedia is borked to Editing and contributions broken on FAMEPedia.Oct 17 2021, 22:49
Unknown Object (User) added a comment.EditedOct 18 2021, 04:28

This happens on multiple wikis. Likely Caused by running compressOld.

What might be the cause though?

In T8184#165079, @Ugochimobi wrote:

What might be the cause though?

Likely Caused by running compressOld.

Upstream ticket was filed

John raised the priority of this task from High to Unbreak Now!.Oct 18 2021, 17:59

Meets criteria of total block on any editing activity

Unknown Object (User) added a comment.Oct 18 2021, 18:05

https://phabricator.wikimedia.org/T39882#410958

Seems to have happened sometimes when running compressOld, in MediaWiki, since 1.18/1.19 at least.

I have a rough understanding of the potential problem, I'm going to try and create a script to hopefully identify actual troublesome revisions.

Unknown Object (User) added a comment.Oct 18 2021, 18:42
universalomega@mwtask1:~$ sudo -u www-data php /srv/mediawiki/w/maintainance/storage/checkStorage.php --wiki=famepediawiki
Checking...
Error: Incorrect object header for concat bulk row 99 in old_id 1465, revisions 2508, 2509
Error: Incorrect object header for concat bulk row 201 in old_id 1452, revisions 2483, 2484
Error: Incorrect object header for concat bulk row 202 in old_id 1453, revisions 2485, 2486
Error: Incorrect object header for concat bulk row 209 in old_id 1455, revisions 2489, 2490
Error: Incorrect object header for concat bulk row 201 in old_id 19407, rev_id 26860
Error: Incorrect object header for concat bulk row 16785 in old_id 19464, rev_id 26976
Error: Incorrect object header for concat bulk row 209 in old_id 19477, revisions 27002, 27003
Error: Incorrect object header for concat bulk row 209 in old_id 22495, rev_id 30245
Error: Incorrect object header for concat bulk row 72 in text rows 24014, 24015, 24016, 24017, 24018, 24019, 24020, 24021, 24022, 24023, 24024, 24025, 24026, 24027, 24028, revisions 31856, 31857, 31858, 31859, 31860, 31861, 31862, 31863, 31864, 31865, 31866, 31867, 31868, 31869, 31870, 31871
Error: Incorrect object header for concat bulk row 50 in text rows 24634, 24635, 24636, 24637, 24638, 24639, 24640, 24641, 24642, 24643, 24644, 24645, 24646, 24647, revisions 32545, 32546, 32547, 32548, 32549, 32550, 32551, 32552, 32553, 32554, 32555, 32556, 32557, 32558, 32559, 32560
Error: Incorrect object header for concat bulk row 99 in text rows 1465, 24472, 24473, 24474, 24475, 24476, 24477, 24478, 24479, 24480, 24481, 24482, 24483, 24484, 24485, revisions 32363, 32344, 32345, 32346, 32347, 32348, 32349, 32350, 32351, 32352, 32353, 32354, 32355, 32356, 32357, 32358
Error: Incorrect object header for concat bulk row 201 in text rows 25975, 25976, 25977, 25978, 25979, 25980, 25981, 25982, 25983, 25984, 25985, 25986, 25987, revisions 33955, 33956, 33957, 33958, 33959, 33960, 33961, 33962, 33963, 33964, 33965, 33966, 33967
Error: Incorrect object header for concat bulk row 19470 in text rows 25766, 25767, 25768, 25769, 25770, 25771, 25772, 25773, 25774, 25775, 25776, 25777, 25778, 25779, 25780, 25781, 25782, revisions 33719, 33720, 33721, 33722, 33723, 33724, 33725, 33726, 33727, 33728, 33729, 33730, 33731, 33732, 33733, 33734, 33735, 33736
Error: Incorrect object header for concat bulk row 202 in text rows 1453, 26033, 26034, 26035, 26036, 26037, 26038, 26039, 26040, 26041, 26042, 26043, 26044, 26045, revisions 34045, 34019, 34020, 34021, 34022, 34023, 34024, 34025, 34026, 34027, 34028, 34029, 34030, 34031, 34032
Error: Incorrect object header for concat bulk row 209 in text rows 26105, 26106, 26107, 26108, 26109, 26110, 26111, 26112, 26113, 26114, 26115, 26116, revisions 34107, 34108, 34109, 34110, 34111, 34112, 34113, 34114, 34115, 34116, 34117, 34118
Error: Incorrect object header for concat bulk row 27981 in text rows 28589, 28590, 28591, 28592, 28593, 28594, 28595, 28596, 28597, 28598, 28599, 28600, 28601, 28602, 28603, 28604, 28605, 28606, revisions 36917, 36918, 36919, 36920, 36921, 36922, 36923, 36924, 36925, 36926, 36927, 36928, 36929, 36930, 36931, 36932, 36933, 36934
Error: Incorrect object header for concat bulk row 16785 in text rows 29343, 29345, revisions 37718, 37720


Errors:
Damaged text, need to be restored from a backup: 2508,2509,2483,2484,2485,2486,2489,2490,26860,26976,27002,27003,30245,31856,31857,31858,31859,31860,31861,31862,31863,31864,31865,31866,31867,31868,31869,31870,31871,32545,32546,32547,32548,32549,32550,32551,32552,32553,32554,32555,32556,32557,32558,32559,32560,32363,32344,32345,32346,32347,32348,32349,32350,32351,32352,32353,32354,32355,32356,32357,32358,33955,33956,33957,33958,33959,33960,33961,33962,33963,33964,33965,33966,33967,33719,33720,33721,33722,33723,33724,33725,33726,33727,33728,33729,33730,33731,33732,33733,33734,33735,33736,34045,34019,34020,34021,34022,34023,34024,34025,34026,34027,34028,34029,34030,34031,34032,34107,34108,34109,34110,34111,34112,34113,34114,34115,34116,34117,34118,36917,36918,36919,36920,36921,36922,36923,36924,36925,36926,36927,36928,36929,36930,36931,36932,36933,36934,37718,37720

Flag statistics:
object,utf-8                        14657 33.44%
utf-8                               29174 66.56%

Local object statistics:
concatenatedgziphistoryblob          3032 20.69%
historyblobstub                     11625 79.31%

checkStorage.php can resolve this with an XML dump. So hopefully, this will only require someone from the MW team to loop through all XML dumps with the checkStorage script

Unknown Object (User) added a comment.Oct 19 2021, 06:56

https://gerrit.wikimedia.org/r/731819 will fix the checkStorage script. I had to make that change locally on mwtask1 last time to run it, which we can do again if necessary.

Unknown Object (User) added a comment.EditedOct 19 2021, 22:08

Here's a script that can loop over backups, running --fix on checkStorage.php:

#! /bin/bash 

usage(){
        echo "Usage: sh fixStorage.sh /path/to/backups/directory"
        exit 1
} 

if [ ! -d $1 ]; then
        echo "directory does not exist!"
        usage
        exit 1
fi 

for backup in $(ls $1);
do
        wiki=`echo $backup | cut -d"." -f1`

        echo "Running checkStorage.php --fix $backup for $wiki"
        sudo -u www-data /usr/bin/php /srv/mediawiki/w/maintenance/storage/checkStorage.php --wiki=$wiki --fix $backup
done

https://github.com/miraheze/mediawiki/pull/3802 fixes checkStorage.php

Unknown Object (User) added a comment.EditedOct 21 2021, 03:37

I was able to finish compiling mwdumper and got that correctly working, after a few modifications to it (https://gerrit.wikimedia.org/r/732699), but now I have one final fatal in checkStorage.php, that I'm unsure with:

[aa5196c5f29f9d4d8282d8e1] [no req]   MediaWiki\Revision\RevisionAccessException: No such slot: 3
Backtrace:
from /srv/mediawiki/w/includes/Revision/RevisionSlots.php(114)
#0 /srv/mediawiki/w/includes/Revision/RevisionSlots.php(95): MediaWiki\Revision\RevisionSlots->getSlot(integer)
#1 /srv/mediawiki/w/includes/import/WikiRevision.php(495): MediaWiki\Revision\RevisionSlots->getContent(integer)
#2 /srv/mediawiki/w/maintenance/storage/checkStorage.php(518): WikiRevision->getContent(integer)
#3 /srv/mediawiki/w/includes/import/WikiImporter.php(571): CheckStorage->importRevision(WikiRevision, WikiImporter)
#4 /srv/mediawiki/w/includes/import/WikiImporter.php(1059): WikiImporter->revisionCallback(WikiRevision)
#5 /srv/mediawiki/w/includes/import/WikiImporter.php(926): WikiImporter->processRevision(array, array)
#6 /srv/mediawiki/w/includes/import/WikiImporter.php(861): WikiImporter->handleRevision(array)
#7 /srv/mediawiki/w/includes/import/WikiImporter.php(678): WikiImporter->handlePage()
#8 /srv/mediawiki/w/maintenance/storage/checkStorage.php(513): WikiImporter->doImport()
#9 /srv/mediawiki/w/maintenance/storage/checkStorage.php(371): CheckStorage->restoreText(array, string)
#10 /srv/mediawiki/w/maintenance/storage/checkStorage.php(35): CheckStorage->check(boolean, string)
#11 {main}
Unknown Object (User) added a comment.Oct 21 2021, 07:00

https://gerrit.wikimedia.org/r/732342 is another patch that was necessary to do on mwtask1 today. Thought it'd be a good idea to document that here.

I was able to finish compiling mwdumper and got that correctly working, after a few modifications to it, but now I have one final fatal in checkStorage.php, that I'm unsure with:

[aa5196c5f29f9d4d8282d8e1] [no req]   MediaWiki\Revision\RevisionAccessException: No such slot: 3
Backtrace:
from /srv/mediawiki/w/includes/Revision/RevisionSlots.php(114)
#0 /srv/mediawiki/w/includes/Revision/RevisionSlots.php(95): MediaWiki\Revision\RevisionSlots->getSlot(integer)
#1 /srv/mediawiki/w/includes/import/WikiRevision.php(495): MediaWiki\Revision\RevisionSlots->getContent(integer)
#2 /srv/mediawiki/w/maintenance/storage/checkStorage.php(518): WikiRevision->getContent(integer)
#3 /srv/mediawiki/w/includes/import/WikiImporter.php(571): CheckStorage->importRevision(WikiRevision, WikiImporter)
#4 /srv/mediawiki/w/includes/import/WikiImporter.php(1059): WikiImporter->revisionCallback(WikiRevision)
#5 /srv/mediawiki/w/includes/import/WikiImporter.php(926): WikiImporter->processRevision(array, array)
#6 /srv/mediawiki/w/includes/import/WikiImporter.php(861): WikiImporter->handleRevision(array)
#7 /srv/mediawiki/w/includes/import/WikiImporter.php(678): WikiImporter->handlePage()
#8 /srv/mediawiki/w/maintenance/storage/checkStorage.php(513): WikiImporter->doImport()
#9 /srv/mediawiki/w/maintenance/storage/checkStorage.php(371): CheckStorage->restoreText(array, string)
#10 /srv/mediawiki/w/maintenance/storage/checkStorage.php(35): CheckStorage->check(boolean, string)
#11 {main}

@John Do you have any idea about this?

Unknown Object (User) added a comment.Oct 21 2021, 20:21

@John Do you have any idea about this?

https://gerrit.wikimedia.org/r/732702 will fix that issue I believe

Unknown Object (User) added a comment.EditedOct 21 2021, 23:03

I have fixed famepediawiki, testwiki, and test3wiki now. I can probably take over and do this on all wikis now, if that is wanted. But it would be easier for Reception123 to, as I would have to copy all XML files to my home directory to extract them. For now, only famepediawiki seemed to be blocking all edits, and at least that one is fixed.

Here is the script again, which I made specifically for this. It should loop over all files in a specified directory, running checkStorage.php --fix with them:

#! /bin/bash

usage(){
        echo "Usage: bash fixStorage.sh /path/to/backups/directory"
        exit 1
}

if [ ! -d $1 ]; then
        echo "Directory does not exist!"
        usage
        exit 1
fi

/usr/local/bin/logsalmsg "Started running checkStorage.php --fix on all wikis"

for backup in $(ls $1);
do
        wiki=`echo $backup | cut -d"." -f1`

        sudo -u www-data gunzip $1/$backup
        sleep 2

        echo "Running checkStorage.php --fix $wiki.xml for $wiki"

        sudo -u www-data /usr/bin/php /srv/mediawiki/w/maintenance/storage/checkStorage.php --wiki=$wiki --fix $1/$wiki.xml
        sleep 2

        sudo -u www-data rm -f $1/$wiki.xml
        sleep 2
done

/usr/local/bin/logsalmsg "Finished running checkStorage.php --fix on all wikis"
Unknown Object (User) lowered the priority of this task from Unbreak Now! to High.Oct 22 2021, 05:55

@Universal_Omega will be working tomorrow (his time) to make the script also unzip and delete files upon running the script as there's not enough space for unzipped files

Unknown Object (User) claimed this task.EditedOct 22 2021, 19:02

The above script is now in progress, running checkStorage.php --fix on all wikis.

Unknown Object (User) moved this task from Backlog to Short Term on the Technology-Team (MediaWiki) board.Oct 22 2021, 19:02
Unknown Object (User) moved this task from Backlog to Maintenance Script Run on the MediaWiki board.
Unknown Object (User) moved this task from Unsorted to Short Term on the Universal Omega board.
Unknown Object (User) closed this task as Resolved.Oct 23 2021, 03:04

Script has finished, it seemed to run fixes on 149 wikis.