When to Use (or Not Use) Runpod's Proxy

Runpod uses a proxy system to ensure that you have easy accessibility to your pods without needing to make any configuration changes. This proxy utilizes Cloudflare for ease of both implementation and access, which comes with several benefits and drawbacks. Let's go into a little explainer about specifically how the Runpod proxy works and when it's most appropriate to use - and when you may want to bypass it.

How the proxy works

The proxy ensures that you always have the same method of accessing a pod on a given port, no matter what networking changes might occur, and that format is pod-ID, port, and then proxy.runpod.net, e.g. in the following format:

https://s7breobom8crgs-3000.proxy.runpod.net/

So for this ComfyUI pod that is defaulted to accept http requests on port 3000, no matter what changes, you can plug in that URL and it will always work. While Secure Cloud pod IP addresses generally should not change very often, there are situations where it happens, often due to network maintenance. Community Cloud IP pod addresses are liable to change at the discretion of the host, so those are not nearly as set in stone.

Drawbacks of the proxy

There are two main drawbacks to the proxy that should be considered:

1. As with any proxy, adding more hops increases latency and the potential for network interruptions. Even under perfect network conditions, you're still bound by the speed of light, and all other things being equal, making the data stream traverse a larger physical distance will add at least a few milliseconds based solely on that. Real world situations, it often ends up being more than that.
2. Cloudflare has a default timeout of approximately 100 seconds if the connection is not kept alive through some external means. So if you send an API request that simply waits for a response with no further communication, if that request takes longer than 100 seconds to fulfill it will time out. This can especially be an issue with very large hosted LLMs working on large context bodies of work, or intensive video or image generation. If the request starts streaming within 100 seconds that is fine, but if it doesn't even start the initial stream by then, nothing will be received at all as the connection will be closed.

How to get around using the proxy

Most official Runpod templates are set up to use the proxy. But if you'd rather decline the use of it, here's how to do that:

1. Edit the template to switch the HTTP exposed ports to TCP. Note that you cannot expose the same ports on both HTTP and TCP.

2. Go to the Connect -> TCP Port Mapping screen to find out what the IP and ports are for the pod. The system will map random external port numbers to exposed ports at random - at this time there is no way to define specific ports to always be used.

3. Since in this case the pod is a ComfyUI pod, you can just plug in the IP and the assigned external port to access it.

If you need to define the IP address and port in code, naturally, it's going to be easiest to maintain if you set a variable for the URL, rather than hard-coding it.

Conclusion

So the upshot is, the proxy is genuinely useful to streamline and standardized pod template and give a way to access them easily by smoothing out wrinkles that are natural in networking, but there's also a lot of reasons why you may not want to use it. Ultimately, you have to decide what's right for you, but we want to give you the tools to complete the workflows you want them to be completed.

Questions? Feel free to run them by our Discord!

‍

Author profile: Brendan McKeag