Spaces:

John6666

/

civitai_to_hf

like 12

Running

feature to download multiple files

I put it on, but I don't know if this is the right way to do it.

can you add lycoris,dora tab? or what more tabs? it will be very helpful to bulk download or brute force download all content from a user profile.

It seems that there is no distinction between Lycoris and LoRA within Civitai. I'm just filtering things like Pony and Flux on my own, so I can add as many items as I like. Is that one okay?
https://github.com/civitai/civitai/wiki/REST-API-Reference

enum[] (Checkpoint, TextualInversion, Hypernetwork, AestheticGradient, LORA, Controlnet, Poses)

for example this user profile user/Konoko, has 99 loras, but most are lycoris, the downloader only finds 23 according to the tab so it only lets you select and download 23, lycoris are similar to lora but they are not lora, some profiles use lots of lycoris and doras, similar variants of loras , its still being in use as i have been seen Lycoris and Doras for Illustrous models

That application seems like it would be useful for my implementation. I'll give it a try. Well, if I can just figure out the exact name of them, I can just write that. It takes a little time to find the correct name. But that's all.

It was a hidden item. Or rather, the Civitai API manual had just not been updated.
I've updated space. If you check LoCon, you can only search for LyCoris. I didn't change the name because that's how it is internally in Civitai.

It was a hidden item. Or rather, the Civitai API manual had just not been updated.
I've updated it. If you check LoCon, you can only search for LyCoris. I didn't change the name because that's how it is internally in Civitai.

working pretty well, from a profile with 20 i could download 15, 5 had the tag Other, there is no tab for other i wonder why civitai has that option but many profile have it, i think the tab other next to poses tab will be fine

i think the tab other next to poses tab will be fine

I added now. I think it's just that the person in charge of updating the API manual hasn't been decided yet, and they've forgotten about it. Anyway, it looks like it hasn't been updated for a long time. Well, there are similar phenomena on HF too.

pretty good, but not sure if a glitch,bug,error but no matter the option or profile, now it shows No item found

There is one known bug or cut-and-paste specification. I'll fix it tomorrow or the day after.
If you duplicate it for yourself, it should be less likely to happen.

There is one known bug or cut-and-paste specification. I'll fix it tomorrow or the day after.
If you duplicate it for yourself, it should be less likely to happen.

didnt work duplicating it but i will wait for a fix, thank you so much for the updates and help

I think I've fixed the known bugs I mentioned above, but there's always the possibility of other bugs.
The slow search speed of Civitai is probably just the way Civitai is designed...

its working now, but 2 bugs, if you select all tabs it will null all selections cause motion tab is bugged, if you unselect motion it will work normally

Oh. Thank you. The cause has been identified and corrected. It seems that Motion is registered under the name MotionModule.

im trying to download all loras from a user profile with 500+ models, https://civitai.com/user/bolero537/, with all tabs on, only finds 233 model with newest tab and 261 with highest rated, sometimes after downloading 200+ models, it gives error, but changing the civitai key to reset the download cap from civitai helps

The Civitai API is not very good for continuous downloading. For example, it is possible to specify multiple Civitai keys, but I'm sleepy today, so I'll do it tomorrow.😪
For now, I've added all the known tabs, but this is probably a bug in the Civitai API. It's strange that the number changes depending on the sorting method.

For now, I've modified it so that if an error occurs while downloading multiple files, a list of the files that have not yet been downloaded is displayed.
What should I do about the API key for Civitai...?
It's difficult to tell for sure whether the key is causing the problem or not, so I think you could use a different key if it fails, or use multiple keys in order from the beginning, but it seems like it would be difficult to use.
I wonder if you could avoid errors by setting a cool-down period between downloads.

What kind of method would be convenient?

The Civitai API can only search for up to 100 items at a time (model page equivalent), so I added a function to automatically crawl and retrieve all of them. If you set the page to 0, it will go and retrieve all of them.
When I tried it with bolero537, it went over 1000. I often download his LoRA and LyCoris too.
By the way, even if you don't select all the tabs, if you unselect the tabs, it will be treated the same as if you had selected all of them.

Edit_ Nevermind, i think its fixed ,it was just a bug, working good again

////works very good so far but im struggling with some accounts with 1 year old content or with 2 years old, probably add Old or All to sort tabs, from user https://civitai.com/user/advokat i can only get 69 links and there are probably more downloadable files,maybe 100+

It seems to have been resolved, but I'll write it down anyway in case it's useful to someone.

probably add Old or All to sort tabs

That sort is something that is passed to the Civitai API, so if I change it on my own, it won't work. I think AllTime was there.
It's possible for me to add individual sorting, but in this case, the problem is not the order in which they are lined up, but that they can't be obtained.

I think i found the bug, im trying to download this flux model with two versions fp16 and fp8 https://civitai.com/models/978314/ultrareal-fine-tune, but its only downloading the version f16, the same is happening with old models like https://civitai.com/models/9052/lofi, the problem is that the versions share a name it will only download 1 version, idk if the downloaded can rename or download both even if the name is the same

Thanks for the report. I think I understand the situation. It's a Civitai specification, but it's a bug in the end. I think it can be fixed here. In short, the same file name is specified in Civitai, so it gets overwritten...😅
I'm going to think of a way to avoid it and apply it.

Fixed. There was one problem I thought there would be, and there was another problem. The information in the Civitai API manual was out of date again...
Anyway, I improved the behavior when there are multiple files. I also added an on/off switch for automatic renaming.

I wanted to update a repo with new files but it created a copy of every file,i manually downloaded ultrarealFineTune_v10.safetensors 22gb version and ultrarealFineTune_v10.safetensors 11gb version and renamed this one to ultrarealFineTune_v10_1.safetensors, but it created two copies of the 22gb version and a copy of all files, its probably not verifying if the same file with size name exists, if the size varies it should rename it

Hmm, because of the performance issues with the free Spaces VM, I can only judge by the file name, so it's a bit of a hassle, but I think I'll try to look at the file hash.

Sorry, there was a bug in aria2c that caused it to fail when there were query parameters. It wasn't a bug in aria2c itself, but a problem with forgetting to quote. I've fixed that.
I'm going to make the existing file check smarter. I wish it could be done with a lighter process...

Edit:
I made it.

Its giving an error message when downloading files links,

its also dragging all training data duplicating the links

Instead of choosing between uploading models or datasets

We need a button to include or not datasets before the process of downloading links

Its giving an error message when downloading files links,

It was a mysterious bug in aria2c that prevented normal URLs from being downloaded when quoted... what the heck

We need a button to include or not datasets before the process of downloading links

I added it (checkbox for Civitai API's internal value).

i think civitai updated something yesterday, im trying to get files form this profile https://civitai.com/user/WRATHGODDESS/models, but i can only get 8 download links, the same has been happening with some profiles

I was able to confirm the symptoms here too. The number of models that can be seen decreases when you are logged out of Civitai. It doesn't seem to be based on the criterion of being nsfw. It seems like the problem could be solved by passing the API key when searching, but I don't know what to do.
I think it's probably a bug in Civitai... well, I'll try to pass it.

Fixed.

john6666's spaces not loading, temporal bug or problem?

I've been nuked by stepping on a mysterious mine space with my main account. I think it'll heal eventually, but there's nothing I can do.🥶

My main account has been restored,😀 but there are still some parts that don't seem to be working properly.
Anyway, I guess I'll have a few days to get my homework done.