provider/irdma: Implement CoCo buffer APIs

[10000TB]

Implement CoCo DMA Bounce buffer APIs defined in #1748

• alloc parent domain
• ibv_alloc_buf, ibv_free_buf
• migration to use verbs_qp in order to be able to use *{qp, cq, srq}_ex{2} cmds
• irdma-abi.h updated to include irdma driver specific ioctl attributes.
• Add a ibv_cmd_create_srq_ex2 to libibverbs to support irdma srq creation via ioctl.
• Migrations
  • QP (push dbl / hw state included)
  • CQ
  • SRQ (shadow area appended at the end for doorbell)
• irdma driver support ( Review in Progress )
• Kernel Integrations.

cc: @jakemoroni @jpirko

[jakemoroni]

Hi,

Thanks for getting a jump on this. We should probably work out some of the uAPI details since irdma is a bit unusual in that regard.

As you are aware, irdma handles user-allocated QP/CQ/SRQ ring buffers differently from other drivers (from what I can tell). For irdma, QP/CQ/SRQ creation is a two step process where the ring buffers are allocated and registered as special memory regions (indicated via flags in udata) using the normal reg_mr command prior to issuing the actual ibv_cmd_create_qp/cq/srq call. The driver maintains an internal registry to correlate these special MRs with their child ring object when the object is created.

So, I think we have a few options:

1. Add a flag attribute to irdma's reg_dmabuf_mr command (UVERBS_ATTR_FLAGS_IN) to indicate whether the region is a normal memory region or a user ring region, similar to what is passed for reg_mr today. Then, the provider would follow mostly the same flow, but would just use reg_dmabuf_mr for dmabuf backed rings. This is likely the simplest change in terms of the irdma kernel driver as well since it already shares a lot of code between reg_dmabuf_mr and normal reg_mr.
2. Change irdma to support providing umem during the actual create_qp/cq/srq call. This is similar to what you've done but would ideally use the new UVERBS_ATTR_UMEM attribute as opposed to adding new fields to the driver udata for each command. This would be a larger change and should probably be treated as a separate endeavor if we decide to go down this route, since it's kind of unrelated to dmabuf ring support (though we would get that for free).
3. There were also talks of some way to allow for mmap-ed dmabuf VAs to be passed to normal ibv_reg_mr. I don't know if that's still PoR but if so, then we wouldn't need any irdma kernel driver changes at all.

Option 1 is the easiest and probably carries less risk, but I am not sure if the maintainers would prefer option 2 since it's a more standard way.

As a side note, we will probably need to fix a few irdma kernel driver issues before adding new features:

1. Need to support robust udata. I just sent a patch series for this.
2. There were some bugs related to this two step user ring process. Specifically, we need to prevent a bad userspace from deregistering these special MRs if there's still a QP/CQ/SRQ object in HW pointing to it. FWIW, this probably wouldn't exist with option 2, but we're still stuck supporting the two step process anyway.

[10000TB]

SG to clarify and align on uAPIs. Good call.

The simpler the better - Therefore, I also prefer option 1.

Option 2 appear a reasonable step forward to converge with standard ways. Though, non-trivial work to support IOCTl ABIs appears needed. Simply modifying the legacy write syscall ABI structs appears a plausible middle ground. A recent patch for fixing a vulnerability on user triggered NULL deref appears specific to the two step process and a counter argument against the two step process (I think your side is also spot on). Though I don't think the two step process is bad. Converging with standard ways is probably the strongest argument, if the alignment is option 2.

For VA dmabuf, it sounds interesting. I'd be curious to learn more. My limited understanding is that kernel side needs a dmabuf fd to cast it to dmabuf structure. Maybe there is no need to cast it at all in kernel with the mmapped approach.

[10000TB]

I am prepping ioctl support for irdma driver. Will refresh this shortly.
I'll also need gather feedback from, and align with other folks on my end, like @jakemoroni.
For now, keeping this as draft.

Regards,
David

[10000TB]

Hello,

kernel RDMA/irdma series is ready (being reviewed internally). Will send them out shortly.

Regards,
David