Page MenuHomeMiraheze

Allow direct download from link in ImportDump
Open, LowPublic

Description

Direct uploads on MH are limited and no one will likely be able to upload an XML larger than 250MB.

We should consider allowing users to provide links to their Google Drive or Dropbox folders that contain the XML. Then, there would be an alternate script command for that where it would download the files from there rather than from Swift. gdown should work for Google Drive if it's a single file and curl should apparently work for Dropbox.

This would mean that Phabricator would no longer be needed for any imports as larger imports could be handled like that. In any case, such large uploads are not possible on Phabricator so users would have to use external services anyway.

Ideally we should however figure out a way where SRE can be alerted if the XML is larger than say 5GB.

Edit: To avoid the complications with Google Drive and Dropbox, we can initially just support regular links with CURL so for example if someone inputs a mediafire link that would work with a simple cURL

Event Timeline

Reception123 created this task.
PixDeVl moved this task from Backlog to Features on the ImportDump board.
PixDeVl subscribed.
Reception123 renamed this task from Allow Google Drive/Dropbox links in ImportDump to Allow direct download from link in ImportDump.May 7 2024, 18:49
Reception123 updated the task description. (Show Details)