I had another look at this and I don't see an easy way of scraping either gamershell or gamefront. Both use an unique key which is generated for each download request, making automatic scraping impossible as far as I can see :(
Printable View
I had another look at this and I don't see an easy way of scraping either gamershell or gamefront. Both use an unique key which is generated for each download request, making automatic scraping impossible as far as I can see :(
IzNoGoD just had some problem some days ago, he wanted e.g. to download: http://www.gamefront.com/files/5125790
You do the initial http request:
Check 1.txt, you gotta rip two lines:Code:printf "GET /files/5125790 HTTP/1.0\r\nHost: www.gamefront.com\r\n\r\n" | netcat gamefront.com 80 > 1.txt
Second request will use this data:Code:Set-Cookie: PHPSESSID=1mcbhc3hv5h4iid1rddfr7vqk6; path=/
<a href="http://www.gamefront.com/files/service/thankyou?id=5125790" class="downloadNow premium-content" id="downloadLink">
Check 2.txt, you gotta rip this:Code:printf "GET /files/service/thankyou?id=5125790 HTTP/1.0\r\nHost: www.gamefront.com\r\nCookie: PHPSESSID=1mcbhc3hv5h4iid1rddfr7vqk6\r\n\r\n" | netcat gamefront.com 80 > 2.txt
And just download it:Code:var downloadUrl = 'http://media1.gamefront.com/pub2/Call_of_Duty_2/Official_Patches/v13/CallofDuty2Patchv1_3.exe?b17f4b620c6cf1393ffa644f10eea151c975f3a1042cbd46c310d68acbb5e5807a00095c2e30b1cb0c60fa18dc133ad4f7284fffc0c0ac5c176306165711ed57a1f1c14a5ba52be7c140b96bd330899e88548e2ad06cffb08c39eacc6099c1645c43ab564e08f60dcf20027ac61c264fb1df6e4510e4ff7a9dfbd6fc381ea077a55203a8dda6aba141b1&ext=.exe';
Would probably be nice just to reconstruct the file structure based on "Call_of_Duty_2/Official_Patches/v13/CallofDuty2Patchv1_3.exe" in final dl link.Code:wget http://media1.gamefront.com/pub2/Call_of_Duty_2/Official_Patches/v13/CallofDuty2Patchv1_3.exe?b17f4b620c6cf1393ffa644f10eea151c975f3a1042cbd46c310d68acbb5e5807a00095c2e30b1cb0c60fa18dc133ad4f7284fffc0c0ac5c176306165711ed57a1f1c14a5ba52be7c140b96bd330899e88548e2ad06cffb08c39eacc6099c1645c43ab564e08f60dcf20027ac61c264fb1df6e4510e4ff7a9dfbd6fc381ea077a55203a8dda6aba141b1&ext=.exe > CallofDuty2Patchv1_3.exe
Thanks, I'll look into it.
Also, about the total filesize, I wouldn't worry too much. According to the latest archived version (11 Aug) of codfiles.com by archive.org, these were the statistics on their webpage:
Site Statistics
Files: 5862
Archive Size: 71.85 GB
Should be easy ;)
LOL that is so wrong. I was a staff member of Filefront for over 8 years, working on adding/removing COD mod files, and I can tell you now, the cache size of CODFiles is nearer 5 Tb of data. It's one of the reasons the site broke - the database was just too big. And so it wouldn't parse properly.
Well, initial indexing of the filefront cod2 files shows approx 1600 files. Assuming the same average size-per-file as the archive.org reported, that should come to around 25gb.
Havent checked cod1/4/5/6/7/8/9/10/11 etc