Steam Download Tools

[Toddy47]

Lol this happened to me this month, decided to download a bunch of steam games without blocking other servers. By the 3rd day in on the months usage we were 95% lol... I have now learnt my lesson as this months gonna be a costly one.

[simcore]

Please see this post guys.

http://www.getsome.co.nz/showthread.php?76096-TelstraClear-is-getting-all-Steamy!&p=1447602#post1447602

[Nigel Bree]

Just to add to this, it seems that Steam is fairly fixed in its notion of whether a product uses HTTP to download or the classic Steam CDN; it appears to be an either-or kinda deal for now.

I've found one game I own on my list which is quite old but appears to be wired to use the newer HTTP CDN, which is Evil Genius. This goes through the newer protocol where it starts the download using a completely different protocol sequence, starting with an HTTP query to Steam that returns a JSON document containing the list of new HTTP servers to use instead of the classic CDN.

I've added some support for these to my filter app; it's a little crude but it works for now. Basically, if the HTTP download is being employed the returned JSON-format list of servers refers to them by name, whereas the classic CDN protocol uses raw binary IP addresses. In addition, the Steam servers in this separate HTTP have DNS names of the form contentserver1.steampowered.com, contentserver2.steampowered.com and the like.

This means that the Steam client has to do an additional DNS query to resolve those new-style content server names to IP addresses before initiating the HTTP download, and so I can filter out the HTTP-mode downloads entirely by blocking DNS resolution of names with that particular pattern; if and when the initial JSON query which starts the product download includes server names from Telstra's CDN then they should come down fine, but since I've been using Evil Genius as a test (and that's a really really old game) it's clear that for now Steam isn't reporting the existence of other servers via its JSON responses and until that happens, it's all or nothing.

[Milkbot]

man that sucks, thanks for the help guys. Least I still get to enjoy skyrim!

[fradazzle]

Would anyone be able to shed any light as to why the firewall rule is letting "http://steam.affinity.net.nz/" download Mass Effect for me?
Here is what I see in TCP View and in my Windows Firewall Settings:

Steam Download Tools

How can I be downloading from the affinity server (IP: 202.50.176.75) when my firewall settings show that anything other than 203.167.129.4 is blocked? What am I doing wrong?

EDIT: Even if I set up a new rule that specifically blocks the IP address for the affinity server (202.50.176.75), it still slips through.

[Citysoldier]

Probably the http system

[fradazzle]

Probably the http system

You're probably right.
However, I can confirm that using Nigel Bree's little application does ensure that Steam only connects to the TC content server. Thanks Nigel!

[Nigel Bree]

I don't believe the reason for your problem was Steam using HTTP; TcpView was showing that the connection is being made on port 27030, which is the classic steam content server protocol, whereas HTTP connections would show either as plain http (for port 80) in TcpView, or as connections to localhost on some other port depending on any personal firewall software being used (some of which redirects all HTTP traffic to a special proxy process to check it for safety).

However, I'm glad to hear the limiter application does the trick for you; "little" and "just works" were my primary goals in making it.

Note that I currently block HTTP downloads from the main Steam servers, but that's a stopgap while the TC steam server doesn't have port 80 open. I'll have to revise the app when that happens and I can observe the Steam client's actual behaviour in action. Until then I can't tell just what kind of encouragement it will *really* need to avoid using different http servers. It may be that the best approach once the TC server is available is to return a fake DNS result for the main Steam servers that points at the TC server, but until that server is really up it's better to pretend to the Steam client that the main http servers don't exist.

[toofast]

Has anyone been able to sucessfully block http downloads? I tried nigel's program, and even thats not doing it for me. Though tbh, i don't think nigel program was doing anything for me. I have currently just blocked all http to steam on what i have identified as http servers to get it to download, but surely there is a more practical solution.

[Nigel Bree]

I can't speak for anyone else but I've definitely blocked HTTP downloads successfully and I regularly re-test all the new versions I make to ensure that it does on the games I have that use the HTTP download system exclusively (Evil Genius, Mass Effect 2)  - the tool definitely does block Steam from HTTP-downloading those games here, and definitely redirects my non-HTTP downloads to TelstraClear exclusively because I depend on it myself.

It's worth pointing out that what steam-limiter does for HTTP is to block DNS lookups; if you've already made the Steam client start an HTTP download, it'll have internally cached those DNS results and won't ask for them again, and so they'll get through. Basically, steam-limiter works best when it's run all the time (which should be tolerable given how tiny its memory load is, especially compared to Steam itself), or at least is running before Steam loads (for the HTTP side of things, anyway).

If the limiter doesn't work *at all* for you, then there'll be a reason why, and I'm happy to work with you to try and diagnose it (but here on these forums probably isn't the place). However, I'd start with making sure you have the latest version (v0.5) installed and then per the "How it works" documentation, using DbgView to see whether you get the "filter loaded", "filter unloaded" messages and the like.

For instance, here's some DbgView output I just captured, running limiter v0.5 blocking a post 27030 download:

Code: [Select]

[9572] wlgwpstmcon01.telstraclear.co.nz=203.167.129.4
[9572] SteamFilter hook attached
[9572] CAPIJobRequestUserStats - Server response failed 2
[9572] Connect redirected
[9572] Connect redirected
[9572] Connect redirected
[9572] Connect redirected
[9572] ExecCommandLine: ""C:\Program Files\Steam\steam.exe" steam://open/downloads "
[9572] ExecSteamURL: "steam://open/downloads"
[9572] CAPIJobRequestUserStats - Server response failed 2

Here's some similar output (edited down a lot, since Steam is quite chatty about all your game licenses and such when starting) from me starting Steam and doing an HTTP download:

Code: [Select]

[12632] appdatacache.cpp (451) : Assertion Failed: gameID == k_uAppIdInvalid || gameID == m_unAppID
[12632] appdatacache.cpp (451) : Assertion Failed: gameID == k_uAppIdInvalid || gameID == m_unAppID
[12632] C:\Program Files\Steam\crashhandler.dll
[12632] C:\Program Files\Steam\steamerrorreporter.exe
[12632] C:\Program Files\Steam\steamerrorreporter.exe
[12632] Starting minidump reporter process
[12632] Failed spawning steam error reporter process.
[12632] appdatacache.cpp (451) : Assertion Failed: gameID == k_uAppIdInvalid || gameID == m_unAppID
[12632] appdatacache.cpp (451) : Assertion Failed: gameID == k_uAppIdInvalid || gameID == m_unAppID
[12632] CHTTPRequestCache took 175 milliseconds to initialize
[12632] wlgwpstmcon01.telstraclear.co.nz=203.167.129.4
[12632] SteamFilter hook attached
[12632] Connect redirected
[...lots of "license added" lines elided here...]
[12632] Error: texture file 'graphics\support_flag_left' does not exist or is invalid
[12632] Error: texture file 'graphics\support_flag_right' does not exist or is invalid
[12632] Error: texture file 'graphics\support_flag_top' does not exist or is invalid
[12632] Error: texture file 'graphics\support_flag_bottom' does not exist or is invalid
[12632] ExecCommandLine: ""C:\Program Files\Steam\Steam.exe" "
[12632] CAPIJobRequestUserStats - Server response failed 2
[12632] System startup time: 11.60 seconds
[12632] Error: texture file 'public\steam_cloudsync' does not exist or is invalid
[12632] CAPIJobRequestUserStats - Server response failed 2
[12632] gethostbyname refused
[12632] gethostbyname refused
[12632] gethostbyname refused
[12632] gethostbyname refused

Basically, the above traces show the hook DLL injecting into Steam and initializing itself, and the "connect redirected" and "gethostbyname refused" messages show it doing its thing to fiddle with Windows Sockets underneath Steam. In the case of the Mass Effect 2 download, because it can't contact any of the HTTP content servers, Steam actually fails the download during the "preparing files for install" phase. If you go through a similar process with DbgView and Steam Limiter (preferably the latest version, v0.5 at the moment) running, start Steam, and don't see text like the above then something's likely up and if you contact me directly via gmail then we can look into this further.

[Solarghost]

Can you please post what the IP addresses are for the NZ Telstra Clear servers for STEAM downloads for the FreysSteamFilter program. I need to update the ini file

kthxbai

[Citysoldier]

203.167.129.004

[simcore]

I can't speak for anyone else but I've definitely blocked HTTP downloads successfully and I regularly re-test all the new versions I make to ensure that it does on the games I have that use the HTTP download system exclusively (Evil Genius, Mass Effect 2)  - the tool definitely does block Steam from HTTP-downloading those games here, and definitely redirects my non-HTTP downloads to TelstraClear exclusively because I depend on it myself.

It's worth pointing out that what steam-limiter does for HTTP is to block DNS lookups; if you've already made the Steam client start an HTTP download, it'll have internally cached those DNS results and won't ask for them again, and so they'll get through. Basically, steam-limiter works best when it's run all the time (which should be tolerable given how tiny its memory load is, especially compared to Steam itself), or at least is running before Steam loads (for the HTTP side of things, anyway).

If the limiter doesn't work *at all* for you, then there'll be a reason why, and I'm happy to work with you to try and diagnose it (but here on these forums probably isn't the place). However, I'd start with making sure you have the latest version (v0.5) installed and then per the "How it works" documentation, using DbgView to see whether you get the "filter loaded", "filter unloaded" messages and the like.

For instance, here's some DbgView output I just captured, running limiter v0.5 blocking a post 27030 download:

Code: [Select]

[9572] wlgwpstmcon01.telstraclear.co.nz=203.167.129.4
[9572] SteamFilter hook attached
[9572] CAPIJobRequestUserStats - Server response failed 2
[9572] Connect redirected
[9572] Connect redirected
[9572] Connect redirected
[9572] Connect redirected
[9572] ExecCommandLine: ""C:\Program Files\Steam\steam.exe" steam://open/downloads "
[9572] ExecSteamURL: "steam://open/downloads"
[9572] CAPIJobRequestUserStats - Server response failed 2

Here's some similar output (edited down a lot, since Steam is quite chatty about all your game licenses and such when starting) from me starting Steam and doing an HTTP download:

Code: [Select]

[12632] appdatacache.cpp (451) : Assertion Failed: gameID == k_uAppIdInvalid || gameID == m_unAppID
[12632] appdatacache.cpp (451) : Assertion Failed: gameID == k_uAppIdInvalid || gameID == m_unAppID
[12632] C:\Program Files\Steam\crashhandler.dll
[12632] C:\Program Files\Steam\steamerrorreporter.exe
[12632] C:\Program Files\Steam\steamerrorreporter.exe
[12632] Starting minidump reporter process
[12632] Failed spawning steam error reporter process.
[12632] appdatacache.cpp (451) : Assertion Failed: gameID == k_uAppIdInvalid || gameID == m_unAppID
[12632] appdatacache.cpp (451) : Assertion Failed: gameID == k_uAppIdInvalid || gameID == m_unAppID
[12632] CHTTPRequestCache took 175 milliseconds to initialize
[12632] wlgwpstmcon01.telstraclear.co.nz=203.167.129.4
[12632] SteamFilter hook attached
[12632] Connect redirected
[...lots of "license added" lines elided here...]
[12632] Error: texture file 'graphics\support_flag_left' does not exist or is invalid
[12632] Error: texture file 'graphics\support_flag_right' does not exist or is invalid
[12632] Error: texture file 'graphics\support_flag_top' does not exist or is invalid
[12632] Error: texture file 'graphics\support_flag_bottom' does not exist or is invalid
[12632] ExecCommandLine: ""C:\Program Files\Steam\Steam.exe" "
[12632] CAPIJobRequestUserStats - Server response failed 2
[12632] System startup time: 11.60 seconds
[12632] Error: texture file 'public\steam_cloudsync' does not exist or is invalid
[12632] CAPIJobRequestUserStats - Server response failed 2
[12632] gethostbyname refused
[12632] gethostbyname refused
[12632] gethostbyname refused
[12632] gethostbyname refused

Basically, the above traces show the hook DLL injecting into Steam and initializing itself, and the "connect redirected" and "gethostbyname refused" messages show it doing its thing to fiddle with Windows Sockets underneath Steam. In the case of the Mass Effect 2 download, because it can't contact any of the HTTP content servers, Steam actually fails the download during the "preparing files for install" phase. If you go through a similar process with DbgView and Steam Limiter (preferably the latest version, v0.5 at the moment) running, start Steam, and don't see text like the above then something's likely up and if you contact me directly via gmail then we can look into this further.

Dude... Epic.

I'd love to test this and make some more video guides.

[hade1]

Possibly the wrong place to post this, but the "Unofficial Telstraclear Steam Server Content List" thread looks to be a necro thread...

Does anyone know if Telstraclear is still updating their NZ content server with latest steam game data?

Have the firewall rule (windows7) mentioned in another thread in place to block non-TCL server traffic. Has served me well in the past but bought a few games off steam in recent weeks and no dice with getting them to download.

Disabled the firewall rule to download a few of the smaller games, but don't really feel like dropping 18gigs worth of data cap to get Kingdoms of Amalur
Reckoning (which is on sale atm). Figured the TCL Steam server would have it as its been out for a few months but no luck so far with trying to download with rule enabled.


Thanks

[Lias]

See discussions about HTTP content..

[hade1]

Thanks for the reply.

For some reason I thought that the issue with HTTP content(from glancing at some posts in other threads) was that it was bypassing the programs set in place to restrict content to the TCL Steam  server and therefore inccuring a metered data cost for telstra customers?

I assumed my issue related to the content (Kingdoms of Amalur) not being on the TCL Steam server as I am getting the "Steam Server Busy" message that was mentioned
on the previous page (and have encountered in the past before games were added to the TCL Server).

Is the HTTP issue something different entirely?

Apologies for lack of current networking/programming knowledge, havent done anything more complicated than set up a router/firewall since the mid 90s

[Lias]

I'm not an epxert but my understanding is as follows.


Valve have two ways of distributing content. The "old" way, which is the type currently supported by the TCL cache server, and a new HTTP system. Most stuff is on the old system, some is on both, and some is only available on the HTTP system. HTTP distributed stuffs can't be obtained via the TCL server. At some point in the future my understanding is Valve will enable caching of the HTTP CDN for the existing servers, but it's not up and running yet.

You can try disabling your firewall rule, installing the game, the reenabling it before it downloads too much of it. You may find you can get some of the game content from the TCL server that way. If it stops downloading as soon as you turn the rule back on, pretty much you can't get it from the TCL cache and will need to burn data to get it from whereever you can.

[simcore]

That's correct, http content is coming from Valve Master Servers and being cached in typical http caches that ISP's set up (a sneaky way of taking advantage of standard caching setups and enabling faster delivery everywhere at no cost for Valve).

The only way to get that title is to download it, which will be metered as it will not be coming from the TelstraClear Steam Content Server (or any other third party content server for that fact).

There are ways to limit http delivery now, but you still won't be able to get it unmetered by any means, so if you want an http delivered title, then it's got to be downloaded sorry.

Valve have confirmed they are working on two things,

1). Change to the Steam client UI so you can force downloads to a single server instead of load balancing (removing the requirement for firewall rules)
2). An update to the Steam Content Server software to allow http delivery from 3rd party content servers.

We're looking at all these things and more right now and figuring out how we can do a better job of keeping people informed and educated.

[simcore]

Revisit the start of the thread for information on Steam-Limiter which appears to be the most effective solution for Windows OS users.

[Jimmah]

Ugh. Looks like Steam is getting a lot converted - and the fix ain't come. I'm getting a lot of titles that won't download unless I open up my filtering rule. Sucks, since a.) I'm using data by doing so. And b.) TCL's Steam content server is literally the ONLY thing that actually performs at near 100mbps. Considering taking the case to TCL since this Warpspeed malarkey is becoming a bit of a joke.

[simcore]

Hey Jimmah, I'm sorry to hear it's an inconvenience for you.

I think it's going to be a critical mass sort of thing, unless Valve truely decide to take their content delivery methods down an entirely different route. They are unable to provide a timeline for any sort of change at this time, however, I am discussing other options with them for providing unmetered access to these http titles through other means.

We'll have to see how things play out on their schedule however.

Cheers.

[Citysoldier]

So lame.  The minute we get unmetered steam they pull this crap on us

[Speakman]

just get a bigger plan

[toofast]

just get a bigger plan

Pretty much. I stayed with TCL for a while with the unmetered, and still struggled to sit inside my 40gb cap. Even with unmetered steam, TCL data plans are still way too little bang for buck. Especially when most providers have unmetered steam, and bigger data plans. TCL really gotta get their head out of their ass.

[Camo260]

Wait so this new system won't allow us to get unmetered access for some games? D:
Damn Ghost Recon Future Soldier, new games have such large downloads and I am getting such low speeds for this lol.
Hope we get unmetered access for everything again.