function has_internet() doesn't work with out proxy

[adrinia]

I'm not that good at this but pxweb stopped working when my job changed dns configuration. I got it working if I changed the function has_internet(). I ran this in my session:
assignInNamespace("has_internet", function(host) TRUE, ns = "pxweb")

It appears that has_internet() s use of nslookup() won't function with our dns configuration since it needs "local lookup". Maybe one can use GET to check internet connection instead of nslookup?

[MansMeg]

Sounds reasonable. Do you want to do a PR?

[adrinia]

Sorry for the late reply. My first suggestion to use GET to check the internet connection is not very elegant when I've thought more about ut. Since GET is used at a later stage when the actual downloading of the data is done. That would just mean use GET and do a complete download to check if there is internet and then afterwards do GET again. Is it possible to completely remove the has_internet function? If there is no internet connection I guess that would be apparent later on anyhow through an error message from GET? Or would it be possible to just output a message that "nslookup was not possible" as a warning but still proceed with the GET?

[MansMeg]

I don't remember exactly why I use nslookup - but I agree it is not a good solution. The point is to check whether the user has access to the internet or not with little cost by pinging the base url. I think something like this could be a solution:

library(httr)

has_internet <- function(url, timeout_sec = 1) {
  tryCatch({
    res <- HEAD(url, timeout(timeout_sec))
    status_code(res) < 400
  }, error = function(e) {
    FALSE
  })
}

has_internet(url)

[adrinia]

It doesn't seem to work. Does it work on your machine? In the case of using url = "api.scb.se" this line doesn't work:
httr::status_code(httr::HEAD("api.scb.se"))

This however does:
httr::status_code(httr::HEAD("https://api.scb.se"))

Unfortunately, even if I do pxweb_interactive("https://api.scb.se") it still doesn't work. Chat GPT suggests that the url gets shortened in this section in pxweb/R

/pxweb_interactive.R

if(!is.null(x)){
    **url_parsed <- parse_url(x)**
    if(parsed_url_has_hostname(url_parsed)){
      if(!has_internet(url_parsed$hostname)){
        message(no_internet_msg(url_parsed$hostname))
        return(NULL)
      }
    }
  }

So even though I'm using https://api.scb.se, has_internet() will get api.scb.se which doesn't work (at least on my machine).

EDIT: in parse_url(x) this happens: url_parsed <- httr::parse_url("https://api.scb.se"). And url_parsed$hostname the becomes "api.scb.se"

[muuankarski]

in ropengov/geofi we have check_api_access()-function that could be of help?

[MansMeg]

I think the has_internet rather checks internet access. Not access to an API.

[MansMeg]

Hmmm. I think I overdid it. I just found this function that should be all that is needed, I think.

curl::has_internet()

A similar problem was identified in 2020 and seem to have been fixed. curl also enables handling proxies etc.
rOpenGov/eurostat#150

Could you try if this works for you, then this should be an easy fix, I think.

[adrinia]

Tried it, works perfectly!

[MansMeg]

Excellent! Ill try to update the package in the next couple of days.

[adrinia]

Wonderful, thank you very much!