Hello! If you are the proud owner of a Golang script currently scraping the hell out of @pypi, you might notice that it has stopped working! @EWDurbin and I would like to have a chat with you! Email us at admin@pypi.org k thx!

10:50 PM · Aug 26, 2020

10
49
7
182
they just need to append a slash to a url in their script 🤣
1
0
0
13
Scraping PyPI with a golang script! Blasphemous! At lease use scrapy to do it!
0
0
0
13
lol... there should be a discord for package server operators.. Had to ping a large tech company recently when their CDN misses racked up >$10k of S3 transfer costs for us. 😞
1
0
0
15
Oof. I just set a monitor for if any of pypi’s miss percentages hop over 3% again. Running public infra for “free” is so easy if you don’t do it reliably... there’s a lot of pain hidden in the hearts of folks who have done it so reliably that people take it for granted.
3
1
0
29
Do you have any restrictions on access that throttle connections? Also, is there any available snapshot of packages one could easily download if analyzing PyPI packages? We do such downloads and having a PyPI snapshot could help you and us as well.
1
0
0
0
the only things we throttle or block are requests that fall back to our backends instead of being served at the edge. the simple index, json documents, and actual file downloads are 100% unthrottled all the time. there is also now a Big Query dataset containing all release data.
1
0
0
4