How do you create a new MemoryCache and populate it with an embedded copy of the dns.json file to prevent downloading the file?

[atc0005]

My goal is to add RDAP query support to a monitoring plugin which will use this library to query the expiration status of a domain. Prototyping indicates that this library does the job well (thank you for providing & maintaining it).

While the check for a single domain won't be run often (e.g., once a day), there may be many executions of the monitoring plugin to cover all domains that are being monitored.

For my use case I wish to embed a copy of the dns.json file at build time and load it into a new MemoryCache. I'd then like to specify that MemoryCache when configuring a client to perform the query. I thought I did that, but the attempt does not appear to be working.

I downloaded a copy of the dns.json file to the same path as a Go file containing this content:

package main

import (
	_ "embed"
	"fmt"
	"log"

	"github.com/openrdap/rdap"
	"github.com/openrdap/rdap/bootstrap"
	"github.com/openrdap/rdap/bootstrap/cache"
)

//go:embed dns.json
var bootstrapRegistryFile []byte

func debug(msg string) {
	fmt.Printf("DEBUG: %s\n", msg)
}

func main() {
	bootstrapCache := cache.NewMemoryCache()
	bootstrapCache.Save("dns.json", bootstrapRegistryFile)

	client := &rdap.Client{
		Bootstrap: &bootstrap.Client{
			Cache: bootstrapCache,
		},

		Verbose: debug,
	}

	req := &rdap.Request{
		Type:  rdap.DomainRequest,
		Query: "example.com",
	}

	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("failed to request RDAP data: %v", err)
	}

	_ = resp

	// ...
}

The output that I'm seeing:

$ go run ./cmd/bugreport/
DEBUG:
DEBUG: client: Running...
DEBUG: client: Request type  : domain
DEBUG: client: Request query : example.com
DEBUG: client: Request URL   : TBD, bootstrap required
DEBUG:   bootstrap: Looking up...
DEBUG:   bootstrap: Question type : dns
DEBUG:   bootstrap: Question query: example.com
DEBUG:   bootstrap: Cache state: dns.json: good
DEBUG:   bootstrap: Downloading dns.json
DEBUG:   bootstrap: Looked up 'example.com'
DEBUG:   bootstrap: Matching entry 'com'
DEBUG:   bootstrap: Service URL #1: 'https://rdap.verisign.com/com/v1/'
DEBUG: client: RDAP URL #0 is https://rdap.verisign.com/com/v1/domain/example.com
DEBUG: client: GET https://rdap.verisign.com/com/v1/domain/example.com
DEBUG: client: status-code=200, content-type=application/rdap+json, length=2440 bytes, duration=235.0072ms
DEBUG: client: Successfully decoded response

For these two lines:

DEBUG:   bootstrap: Cache state: dns.json: good
DEBUG:   bootstrap: Downloading dns.json

The first line is expected: I've just freshly loaded the embedded file into the memory cache.

The second line is not expected: Why isn't the cached copy of the file being used?

This looks to be the relevant portion of the code:

rdap/bootstrap/client.go

Lines 324 to 333 in c9246cb

	if c.registries[registry] == nil || forceDownload {
	c.Verbose(fmt.Sprintf(" bootstrap: Downloading %s", registry.Filename()))
	err := c.DownloadWithContext(question.Context(), registry)
	if err != nil {
	return nil, err
	}
	} else {
	c.Verbose(" bootstrap: Using cached Service Registry file")
	}

I'm struggling to find a way to directly set c.registries[registry] or indirectly do so via exposed functions.

Am I overlooking something obvious?

Thanks in advance for your help.

[wolveix]

Hey @atc0005, thanks for taking the time to open this, and file this issue! This highlighted a bug, which should have been fixed in #55. Please pull the latest version (v0.10.2) and let us know if you run into any issues!

[atc0005]

Hi @wolveix,

You're welcome, and thank you for the quick fix.

After updating github.com/openrdap/rdap v0.10.1 => v0.10.2 I re-ran the example code I provided and got this output:

$ go run ./cmd/bugreport/
DEBUG:
DEBUG: client: Running...
DEBUG: client: Request type  : domain
DEBUG: client: Request query : example.com
DEBUG: client: Request URL   : TBD, bootstrap required
DEBUG:   bootstrap: Looking up...
DEBUG:   bootstrap: Question type : dns
DEBUG:   bootstrap: Question query: example.com
DEBUG:   bootstrap: Cache state: dns.json: good
DEBUG:   bootstrap: Using cached Service Registry file
DEBUG:   bootstrap: Looked up 'example.com'
DEBUG:   bootstrap: Matching entry 'com'
DEBUG:   bootstrap: Service URL #1: 'https://rdap.verisign.com/com/v1/'
DEBUG: client: RDAP URL #0 is https://rdap.verisign.com/com/v1/domain/example.com
DEBUG: client: GET https://rdap.verisign.com/com/v1/domain/example.com
DEBUG: client: status-code=200, content-type=application/rdap+json, length=2440 bytes, duration=265.525ms
DEBUG: client: Successfully decoded response

Thanks also for adding the example of pre-populating the cache:

rdap/bootstrap/client.go

Lines 84 to 90 in b4d5927

	// You can pre-populate the cache with your own copy of a Service Registry file
	// (e.g. one embedded with //go:embed), to avoid downloading it:
	//
	// c := cache.NewMemoryCache()
	// c.Save(bootstrap.DNS.Filename(), embeddedDNSJSON)
	//
	// b := &bootstrap.Client{Cache: c} // Lookup() uses the cached file.

This is great, thank you.

---

A bit of a tangent:

For questions that are usage related and not (suspected of being) bug reports, should we open a new issue or is there another place to discuss use of this library?

For example, I understand that some TLDs are not covered by RDAP yet and WHOIS queries should be performed instead. With the dns.json in the memory cache I'd be interested in how I can reliably query that and if a specific error is returned fallback to performing a WHOIS query. I suspect that this library already exposes what is needed to perform that task, I just haven't put the pieces together yet. For this scenario opening an issue seems to put more of a workload on you & other maintainers vs having someone from the community point us in the right direction.

I see on #38 that someone proposed a mailing list. Was that setup? If not, could enabling GitHub Discussions for this repo work?

[atc0005]

For example, I understand that some TLDs are not covered by RDAP yet and WHOIS queries should be performed instead. With the dns.json in the memory cache I'd be interested in how I can reliably query that and if a specific error is returned fallback to performing a WHOIS query. I suspect that this library already exposes what is needed to perform that task, I just haven't put the pieces together yet.

To answer my earlier question (mostly posed as an example of a usage question others might ask on a mailing list or forum) for anyone who might look for this in the future, the following seems like it would work. It's an incomplete implementation, (just a variation of my earlier example code) but it should work for illustration:

package main

import (
	_ "embed"
	"fmt"
	"log"

	"github.com/openrdap/rdap"
	"github.com/openrdap/rdap/bootstrap"
	"github.com/openrdap/rdap/bootstrap/cache"
)

//go:embed dns.json
var bootstrapRegistryFile []byte

func debug(msg string) {
	fmt.Printf("DEBUG: %s\n", msg)
}

func main() {
	domainQuery := "mit.edu"

	bootstrapCache := cache.NewMemoryCache()
	bootstrapCache.Save(bootstrap.DNS.Filename(), bootstrapRegistryFile)

	client := &rdap.Client{
		Bootstrap: &bootstrap.Client{
			Cache: bootstrapCache,
		},

		Verbose: debug,
	}

	question := &bootstrap.Question{
		RegistryType: bootstrap.DNS,
		Query:        domainQuery,
	}

	answer, err := client.Bootstrap.Lookup(question)
	if err != nil {
		log.Fatalf("failed to lookup RDAP base URLs : %v", err)
	}

	switch {
	case len(answer.URLs) > 0:
		fmt.Printf("RDP servers found for '%s', performing RDAP query ...\n", domainQuery)

		// Perform RDAP query since we have RDAP base URLs to work with.
		req := &rdap.Request{
			Type:  rdap.DomainRequest,
			Query: domainQuery,
		}

		resp, err := client.Do(req)
		if err != nil {
			log.Fatalf("failed to request RDAP data: %v", err)
		}

		// Satisfy build requirements ...
		_ = resp

		// ...

	default:
		// Attempt WHOIS query WHOIS since RDAP servers are not available
		fmt.Printf("No RDP servers found for '%s', performing WHOIS query instead ...\n", domainQuery)

		// ...
	}
}

Output if setting domainQuery to google.com:

$ go run ./cmd/bugreport/
RDP servers found for 'google.com', performing RDAP query ...
DEBUG:
DEBUG: client: Running...
DEBUG: client: Request type  : domain
DEBUG: client: Request query : google.com
DEBUG: client: Request URL   : TBD, bootstrap required
DEBUG:   bootstrap: Looking up...
DEBUG:   bootstrap: Question type : dns
DEBUG:   bootstrap: Question query: google.com
DEBUG:   bootstrap: Cache state: dns.json: good
DEBUG:   bootstrap: Using cached Service Registry file
DEBUG:   bootstrap: Looked up 'google.com'
DEBUG:   bootstrap: Matching entry 'com'
DEBUG:   bootstrap: Service URL #1: 'https://rdap.verisign.com/com/v1/'
DEBUG: client: RDAP URL #0 is https://rdap.verisign.com/com/v1/domain/google.com
DEBUG: client: GET https://rdap.verisign.com/com/v1/domain/google.com
DEBUG: client: status-code=200, content-type=application/rdap+json, length=2822 bytes, duration=314.5542ms
DEBUG: client: Successfully decoded response

Output if setting domainQuery to mit.edu:

$ go run ./cmd/bugreport/
No RDP servers found for 'mit.edu', performing WHOIS query instead ...

I looked at the implementation of this function for guidance:

rdap/client.go

Lines 89 to 95 in b4d5927

	// Do runs the RDAP request req and returns its Response.
	//
	// When req has no server set, the query is bootstrapped to find the
	// authoritative RDAP servers, each of which is tried until one responds
	// successfully. Uninitialised client fields (HTTP, Bootstrap, Verbose) are
	// given defaults on first use.
	func (c *Client) Do(req *Request) (*Response, error) {

specifically:

rdap/client.go

Lines 154 to 157 in b4d5927

	question := &bootstrap.Question{
	RegistryType: *bootstrapType,
	Query: req.Query,
	}

rdap/client.go

Line 163 in b4d5927

answer, err = c.Bootstrap.Lookup(question)

rdap/client.go

Lines 170 to 176 in b4d5927

	// No URLs to query?
	if len(answer.URLs) == 0 {
	return resp, &ClientError{
	Type: BootstrapNoMatch,
	Text: fmt.Sprintf("No RDAP servers found for '%s'", question.Query),
	}
	}

[wolveix]

Hey @atc0005! Great idea regarding discussions, I've enabled them for this repo :)