映射mmio区域写作不起作用[英] Mapping MMIO region write-back does not work

本文是小编为大家收集整理的关于映射mmio区域写作不起作用的处理/解决方法,可以参考本文帮助大家快速定位并解决问题,中文翻译不准确的可切换到English标签页查看源文。

问题描述

我希望所有读写请求都通过CPU缓存来缓存PCIE设备.但是,它无法正常工作.

这些是我对写下MMIO区域的假设.

  1. 写信给PCIE设备仅发生在缓存写入后.
  2. TLP有效载荷的大小是缓存块大小(64b).

但是,捕获的TLP不遵循我的假设.

  1. 写给PCIE设备的每条写入MMIO区域的每条写入.
  2. TLP有效载荷的大小为1B.

i用以下用户空间程序和设备驱动程序将0xff的8字节写入MMIO区域.

用户程序的一部分

struct pcie_ioctl ioctl_control;
ioctl_control.bar_select = BAR_ID;
ioctl_control.num_bytes_to_write = atoi(argv[1]);
if (ioctl(fd, IOCTL_WRITE_0xFF, &ioctl_control) < 0) {
    printf("ioctl failed\n");
}

设备驱动程序的一部分

case IOCTL_WRITE_0xFF:
{
    int i;
    char *buff;
    struct pci_cdev_struct *pci_cdev = pci_get_drvdata(fpga_pcie_dev.pci_device);
    copy_from_user(&ioctl_control, (void __user *)arg, sizeof(ioctl_control));
    buff = kmalloc(sizeof(char) * ioctl_control.num_bytes_to_write, GFP_KERNEL);
    for (i = 0; i < ioctl_control.num_bytes_to_write; i++) {
        buff[i] = 0xff;
    }
    memcpy(pci_cdev->bar[ioctl_control.bar_select], buff, ioctl_control.num_bytes_to_write);
    kfree(buff);
    break;
}

i修改了MTRR,以使相应的MMIO区域写下. MMIO区域从0x0C7300000开始,长度为0x100000(1MB).以下是不同策略的结果.请注意,我使每个区域都独家.

不可接受的

reg00: base=0x080000000 ( 2048MB), size= 1024MB, count=1: uncachable
reg01: base=0x380000000000 (58720256MB), size=524288MB, count=1: uncachable
reg02: base=0x0c0000000 ( 3072MB), size=   64MB, count=1: uncachable
reg03: base=0x0c4000000 ( 3136MB), size=   32MB, count=1: uncachable
reg04: base=0x0c6000000 ( 3168MB), size=   16MB, count=1: uncachable
reg05: base=0x0c7000000 ( 3184MB), size=    1MB, count=1: uncachable
reg06: base=0x0c7100000 ( 3185MB), size=    1MB, count=1: uncachable
reg07: base=0x0c7200000 ( 3186MB), size=    1MB, count=1: uncachable
reg08: base=0x0c7300000 ( 3187MB), size=    1MB, count=1: uncachable
reg09: base=0x0c7400000 ( 3188MB), size=    1MB, count=1: uncachable

写入组合

reg00: base=0x080000000 ( 2048MB), size= 1024MB, count=1: uncachable
reg01: base=0x380000000000 (58720256MB), size=524288MB, count=1: uncachable
reg02: base=0x0c0000000 ( 3072MB), size=   64MB, count=1: uncachable
reg03: base=0x0c4000000 ( 3136MB), size=   32MB, count=1: uncachable
reg04: base=0x0c6000000 ( 3168MB), size=   16MB, count=1: uncachable
reg05: base=0x0c7000000 ( 3184MB), size=    1MB, count=1: uncachable
reg06: base=0x0c7100000 ( 3185MB), size=    1MB, count=1: uncachable
reg07: base=0x0c7200000 ( 3186MB), size=    1MB, count=1: uncachable
reg08: base=0x0c7300000 ( 3187MB), size=    1MB, count=1: write-combining
reg09: base=0x0c7400000 ( 3188MB), size=    1MB, count=1: uncachable

写下

reg00: base=0x080000000 ( 2048MB), size= 1024MB, count=1: uncachable
reg01: base=0x380000000000 (58720256MB), size=524288MB, count=1: uncachable
reg02: base=0x0c0000000 ( 3072MB), size=   64MB, count=1: uncachable
reg03: base=0x0c4000000 ( 3136MB), size=   32MB, count=1: uncachable
reg04: base=0x0c6000000 ( 3168MB), size=   16MB, count=1: uncachable
reg05: base=0x0c7000000 ( 3184MB), size=    1MB, count=1: uncachable
reg06: base=0x0c7100000 ( 3185MB), size=    1MB, count=1: uncachable
reg07: base=0x0c7200000 ( 3186MB), size=    1MB, count=1: uncachable
reg08: base=0x0c7300000 ( 3187MB), size=    1MB, count=1: write-back
reg09: base=0x0c7400000 ( 3188MB), size=    1MB, count=1: uncachable

以下是8B写入不同策略的波形捕获.我已经使用集成逻辑分析仪(ILA)捕获这些波形.请观看pcie_endpoint_litepcietlpdepacketizer_tlp_req_payload_dat设置pcie_endpoint_litepcietlpdepacketizer_tlp_req_valid时.您可以在这些波形示例中计数pcie_endpoint_litepcietlpdepacketizer_tlp_req_valid来计数数据包数.

  1. 不可接受的: link - >正确,正确,1b x 8数据包
  2. 写入组合: link - >正确,8b x 1数据包
  3. write -back : link >意外,1b x 8数据包

系统配置如下.

  • CPU :Intel(r)Xeon(R)CPU E5-2630 V4 @ 2.20GHz
  • OS :Linux内核4.15.0-38
  • pcie设备:xilinx fpga kc705用 litepcie /li>

相关链接

  1. 生成64-byte Read PCIE来自X86 CPU的TLP
  2. 如何实现Intel®架构上的64B PCIE*爆发转移
  3. ryzen支持对内存映射的io(通过pcie界面)吗??
  4. mtrr(内存类型范围寄存器寄存器)控制
  5. patting linux
  6. to tlp:pci express设备如何谈话(第I部分)
简而

如果有人发现可能会上传答案.

我来找约翰·麦卡平(John McCalpin)的文章和答案.首先,不可能映射mmio区域写作.其次,某些处理器可以解决方法.

  1. 映射mmio区域写作不可能

    from以下链接引用/p>

    fyi:WB类型将无法与内存映射IO一起使用.你可以 对位进行编程以将映射设置为WB,但是系统将 一旦获得不知道该如何的交易,它就崩溃 处理.从理论上讲,它可以使用WP或WT进行缓存 从MMIO读取,但必须在软件中处理连贯性.

    报价链接

    仅当我将pat和mtrr设置为WB时,内核崩溃

  2. 在某些处理器上可能有能力解决

    有关缓存访问内存映射IO区域的注释,John McCalpin

    有一组映射至少可以在某些方面使用 X86-64处理器,它基于映射MMIO Space 两次. 用一组允许写入组合的属性绘制MMIO范围 商店(但仅读取).绘制第二次MMIO范围 具有一组允许缓存线读取的属性(但仅

    未上的,非合并的商店).

本文地址:https://www.itbaoku.cn/post/2090991.html

问题描述

I want all read & write requests to a PCIe device to be cached by CPU caches. However, it does not work as I expected.

These are my assumptions on write-back MMIO regions.

  1. Writes to the PCIe device happen only on cache write-back.
  2. The size of TLP payloads is cache block size (64B).

However, captured TLPs do not follow my assumptions.

  1. Writes to the PCIe device happen on every write to the MMIO region.
  2. The size of TLP payloads is 1B.

I write 8-byte of 0xff to the MMIO region with the following user space program & device driver.

Part of User Program

struct pcie_ioctl ioctl_control;
ioctl_control.bar_select = BAR_ID;
ioctl_control.num_bytes_to_write = atoi(argv[1]);
if (ioctl(fd, IOCTL_WRITE_0xFF, &ioctl_control) < 0) {
    printf("ioctl failed\n");
}

Part of Device Driver

case IOCTL_WRITE_0xFF:
{
    int i;
    char *buff;
    struct pci_cdev_struct *pci_cdev = pci_get_drvdata(fpga_pcie_dev.pci_device);
    copy_from_user(&ioctl_control, (void __user *)arg, sizeof(ioctl_control));
    buff = kmalloc(sizeof(char) * ioctl_control.num_bytes_to_write, GFP_KERNEL);
    for (i = 0; i < ioctl_control.num_bytes_to_write; i++) {
        buff[i] = 0xff;
    }
    memcpy(pci_cdev->bar[ioctl_control.bar_select], buff, ioctl_control.num_bytes_to_write);
    kfree(buff);
    break;
}

I modified MTRRs to make the corresponding MMIO region write-back. The MMIO region starts from 0x0c7300000, and the length is 0x100000 (1MB). Followings are cat /proc/mtrr results for different policies. Please note that I made each region exclusive.

Uncacheable

reg00: base=0x080000000 ( 2048MB), size= 1024MB, count=1: uncachable
reg01: base=0x380000000000 (58720256MB), size=524288MB, count=1: uncachable
reg02: base=0x0c0000000 ( 3072MB), size=   64MB, count=1: uncachable
reg03: base=0x0c4000000 ( 3136MB), size=   32MB, count=1: uncachable
reg04: base=0x0c6000000 ( 3168MB), size=   16MB, count=1: uncachable
reg05: base=0x0c7000000 ( 3184MB), size=    1MB, count=1: uncachable
reg06: base=0x0c7100000 ( 3185MB), size=    1MB, count=1: uncachable
reg07: base=0x0c7200000 ( 3186MB), size=    1MB, count=1: uncachable
reg08: base=0x0c7300000 ( 3187MB), size=    1MB, count=1: uncachable
reg09: base=0x0c7400000 ( 3188MB), size=    1MB, count=1: uncachable

Write-combining

reg00: base=0x080000000 ( 2048MB), size= 1024MB, count=1: uncachable
reg01: base=0x380000000000 (58720256MB), size=524288MB, count=1: uncachable
reg02: base=0x0c0000000 ( 3072MB), size=   64MB, count=1: uncachable
reg03: base=0x0c4000000 ( 3136MB), size=   32MB, count=1: uncachable
reg04: base=0x0c6000000 ( 3168MB), size=   16MB, count=1: uncachable
reg05: base=0x0c7000000 ( 3184MB), size=    1MB, count=1: uncachable
reg06: base=0x0c7100000 ( 3185MB), size=    1MB, count=1: uncachable
reg07: base=0x0c7200000 ( 3186MB), size=    1MB, count=1: uncachable
reg08: base=0x0c7300000 ( 3187MB), size=    1MB, count=1: write-combining
reg09: base=0x0c7400000 ( 3188MB), size=    1MB, count=1: uncachable

Write-back

reg00: base=0x080000000 ( 2048MB), size= 1024MB, count=1: uncachable
reg01: base=0x380000000000 (58720256MB), size=524288MB, count=1: uncachable
reg02: base=0x0c0000000 ( 3072MB), size=   64MB, count=1: uncachable
reg03: base=0x0c4000000 ( 3136MB), size=   32MB, count=1: uncachable
reg04: base=0x0c6000000 ( 3168MB), size=   16MB, count=1: uncachable
reg05: base=0x0c7000000 ( 3184MB), size=    1MB, count=1: uncachable
reg06: base=0x0c7100000 ( 3185MB), size=    1MB, count=1: uncachable
reg07: base=0x0c7200000 ( 3186MB), size=    1MB, count=1: uncachable
reg08: base=0x0c7300000 ( 3187MB), size=    1MB, count=1: write-back
reg09: base=0x0c7400000 ( 3188MB), size=    1MB, count=1: uncachable

Followings are waveform captures for 8B write with different policies. I have used integrated logic analyzer (ILA) to capture these waveform. Please watch pcie_endpoint_litepcietlpdepacketizer_tlp_req_payload_dat when pcie_endpoint_litepcietlpdepacketizer_tlp_req_valid is set. You can count the number of packets by counting pcie_endpoint_litepcietlpdepacketizer_tlp_req_valid in these waveform example.

  1. Uncacheable: link -> correct, 1B x 8 packets
  2. Write-combining: link -> correct, 8B x 1 packet
  3. Write-back: link -> unexpected, 1B x 8 packets

System configuration is like below.

  • CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  • OS: Linux kernel 4.15.0-38
  • PCIe Device: Xilinx FPGA KC705 programmed with litepcie

Related Links

  1. Generating a 64-byte read PCIe TLP from an x86 CPU
  2. How to Implement a 64B PCIe* Burst Transfer on Intel® Architecture
  3. Write Combining Buffer Out of Order Writes and PCIe
  4. Do Ryzen support write-back caching for Memory Mapped IO (through PCIe interface)?
  5. MTRR (Memory Type Range Register) control
  6. PATting Linux
  7. Down to the TLP: How PCI express devices talk (Part I)

推荐答案

In short, it seems that mapping MMIO region write-back does not work by design.

Please upload an answer if anyone finds that it is possible.

I came to find John McCalpin's articles and answers. First, mapping MMIO region write-back is not possible. Second, workaround is possible on some processors.

  1. Mapping MMIO region write-back is not possible

    Quote from this link

    FYI: The WB type will not work with memory-mapped IO. You can program the bits to set up the mapping as WB, but the system will crash as soon as it gets a transaction that it does not know how to handle. It is theoretically possible to use WP or WT to get cached reads from MMIO, but coherence has to be handled in software.

    Quote from this link

    Only when I set both PAT and MTRR to WB does the kernel crash

  2. Workaround is possible on some processors

    Notes on Cached Access to Memory-Mapped IO Regions, John McCalpin

    There is one set of mappings that can be made to work on at least some x86-64 processors, and it is based on mapping the MMIO space twice. Map the MMIO range with a set of attributes that allow write-combining stores (but only uncached reads). Map the MMIO range a second time with a set of attributes that allow cache-line reads (but only uncached, non-write-combined stores).