Scripts for converting pytorch grid into L2R'24 COMULISSHGBF Submission Format?

Scripts for converting pytorch grid into L2R'24 COMULISSHGBF Submission Format?  

  By: huboqiang on June 26, 2024, 3:01 a.m.

Dear all

I need some instructions for converting 2D pytorch grid_sample grid into submission nii.gz.

For example, I got a ok result for 0004 with pytorch grid_sample function:

However, the TRE in leaderborad, is 200+:

{
  "name": "COMULISSHGBF",
  "cases": {
    "0004_0000<--0004_0001": {
      "TRE_lm": {
        "mean": 249.23447419979559,
        "detailed": [
          394.1283978539982,
          227.08221950121387,
          358.8919408074519,
          327.3250411567412,
          397.33235645952493,
          220.47409711239033,
          282.25376101368596,
          63.69834172407011,
          249.2802253619205,
          201.04043907684527,
          335.96265768308547,
          114.68517970892336,
          117.92020637623497,
          127.20886199416978,
          175.42450899960602,
          473.5454197726987,
          152.18533998609786,
          282.2592651375634,
          13.94543948063247,
          335.0911858075518,
          244.04301799834022,
          362.49139391768966,
          308.50232584383076,
          197.00175265709328,
          269.08847956352975
        ]
      },
      "LogJacDetStd": {
        "mean": 3.3724719721954794e-06,
        "detailed": 3.3724719721954794e-06
      },
      "num_foldings": {
        "mean": 0.0,
        "detailed": 0.0
      }
    }
}

I mentioned the instructions on the submission page that

When using PyTorch as deep learning framework you are most likely to transform your images with the grid_sample() routine. Please be aware that this function uses a different convention than ours, expecting displacement fields in the format [X, Y, 1, [1, y, x]] and normalized coordinates between -1 and 1. Prior to your submission you should therefore convert your displacement fields to match our convention (see above).

So I write a script to transform pytorch grid into .nii.gz . I think the problem is grid_to_disp:

import sys
import numpy as np
import torch
import nibabel as nib
import torch.nn.functional as F


def create_initial_grid(rtk_module, height, width):
    grid_M = rtk_module().unsqueeze(0)
    grid = F.affine_grid(grid_M, [1, 1, height, width], align_corners=True)
    return grid


def rtk_to_disp(rtk_params):
    rtk_module = RTKModule(rtk_params)
    grid_M = create_initial_grid(rtk_module, 834, 834)
    disp_field = grid_to_disp(grid_M)
    return disp_field.detach().numpy()

def grid_to_disp(grid):
    """
        grid:   (N, H_out, W_out, 2). See torch.nn.functional.grid_sample
        disp:   (N, 2, H, W). See scipy.ndimage.map_coordinates
    """
    H, W = grid.shape[1], grid.shape[2]
    disp_field = (
        (grid / 2)*(torch.tensor([H, W])-1)
    ).flip(-1).float().cpu().permute(1, 2, 0, 3)
    return disp_field


def main():
    l_idxs = [4]
    l_rtks = [
        [-0.05, -0.15, 0.03, 0.0, 0.02],
    ]
    for i_idx in range(len(l_idxs)):
        rtk_params = torch.FloatTensor( np.array(l_rtks[i_idx], dtype=np.float32) )
        disp_field = rtk_to_disp(rtk_params)
        img = nib.Nifti1Image(disp_field.astype(np.float64), np.eye(4))
        nib.save(img, f"./test/disp_{l_idxs[i_idx]:04d}_{l_idxs[i_idx]:04d}.nii.gz")

As there were no test-case for TRE calculation in this challenge, and the submission time were limited, I am looking for your help for debug this code.

Thanks~

Re: Scripts for converting pytorch grid into L2R'24 COMULISSHGBF Submission Format?  

  By: lassehansen on July 15, 2024, 12:03 p.m.

Hi huboqiang,

looks fine to me at first glance. Please send me a link with your Pytorch result so I can check it again here on our site.

Best wishes, Lasse

Re: Scripts for converting pytorch grid into L2R'24 COMULISSHGBF Submission Format?  

  By: huboqiang on July 16, 2024, 6:21 a.m.

Thanks you lassehansen , here is the link. https://www.dropbox.com/scl/fi/z1lfntbj1wgovu6en32zf/test_20240704_v4_pytorch.zip?rlkey=8dglxk81e61c7bud4zls1kffx&st=gl1ht94m&dl=0

Re: Scripts for converting pytorch grid into L2R'24 COMULISSHGBF Submission Format?  

  By: lassehansen on July 16, 2024, 8:57 a.m.

Thanks.

I could align the images with your provided displacement fields using the following code snippet:

disp = torch.load(os.path.join(DISP_PATH, f'grid_{case:04d}_{case:04d}.pt'))
fixed = nib.load(os.path.join(IMG_PATH, f'COMULISSHGBF_{case:04d}_0000.nii.gz')).get_fdata()[:, :, :]
moving = nib.load(os.path.join(IMG_PATH, f'COMULISSHGBF_{case:04d}_0001.nii.gz')).get_fdata()[:, :, 0, 0, :] / 255
fixed_warped = F.grid_sample(torch.from_numpy(fixed).permute(2, 1, 0).unsqueeze(0).float(), disp)[0].permute(2, 1, 0)

So it seems I had to warp the fixed instead of the moving image and in addition had to permute the image dimensions (torch.from_numpy(fixed).permute(2, 1, 0)). Can you check these possible sources of error?

I then assume (untested as I don't have the correct displacement fields, see above) that you are not converting the displacement field but the grid itself. At some point you would need to subtract the identity grid (i.e. something along these lines: disp = disp - F.affine_grid(torch.eye(3)[:2, :].unsqueeze(0), (1, 1, disp.shape[1], disp.shape[2]))).

Let me know if this works for you and feel free to contact us again if there are any further questions or uncertainties. There will also be a Q&A Session on Wednesday, July 17th, 2024, 1:30 PM UTC (https://grand-challenge.org/forums/forum/learn2reg-registration-challenge-449/topic/qa-session-wednesday-july-17th-2024-130-pm-utc-2402/).

Best wishes, Lasse

 Last edited by: lassehansen on July 16, 2024, 8:58 a.m., edited 1 time in total.

Re: Scripts for converting pytorch grid into L2R'24 COMULISSHGBF Submission Format?  

  By: huboqiang on July 17, 2024, 12:39 p.m.

Great Lasse.

I followed your advices and submitted to leader borad. However the TRE is still 50+. So I still need some help.

I put my moving -> fixed pytorch grid in pt file here:

https://www.dropbox.com/scl/fi/wl8h1ww9svnc5vxvnx6qm/meeting_0717.zip?rlkey=he5nrcjrc7ecoti19zj2oq489&st=hgamhk0k&dl=0

nill file were generated using grid_to_disp:

def grid_to_disp(grid):
    """
        grid:   (N, H_out, W_out, 2). See torch.nn.functional.grid_sample
        disp:   (H, W, N, 2). See scipy.ndimage.map_coordinates
    """
    N, H, W, _ = grid.shape
    disp_field = (grid + 1) * torch.tensor([H-1, W-1]).view(1, 1, 1, 2) / 2.0
    base_grid = F.affine_grid(torch.eye(3)[:2].unsqueeze(0), [N, 1, H, W], align_corners=False)
    base_grid = (base_grid + 1) * torch.tensor([H-1, W-1]).view(1, 1, 1, 2) / 2.0
    disp_field = disp_field - base_grid
    return disp_field.permute(1, 2, 0, 3).detach().cpu().numpy()

grid = torch.load(os.path.join(DISP_PATH, f'grid_{case:04d}_{case:04d}.pt'))
disp_field = grid_to_disp(grid)
img = nib.Nifti1Image(disp_field.astype(np.float64), np.eye(4))
nib.save(img, f"./test/disp_{l_idxs[i_idx]:04d}_{l_idxs[i_idx]:04d}.nii.gz")

so could you please give me a correct grid_to_disp ? or you may provide me a example 0004.nill.gz file so I can have a test-case.

Thanks~

 Last edited by: huboqiang on July 17, 2024, 12:42 p.m., edited 1 time in total.

Re: Scripts for converting pytorch grid into L2R'24 COMULISSHGBF Submission Format?  

  By: lassehansen on July 20, 2024, 10:42 p.m.

For your particular case, please try this updated grid_to_disp code.

def grid_to_disp(grid):
    """
        grid:   (N, H_out, W_out, 2). See torch.nn.functional.grid_sample
        disp:   (H, W, N, 2). See scipy.ndimage.map_coordinates
    """
    N, H, W, _ = grid.shape

    grid = grid.permute(0, 2, 1, 3).flip(-1)
    disp_field = grid - F.affine_grid(torch.eye(3, device=grid.device)[:2, :].unsqueeze(0), (1, 1, H, W))
    disp_field = ((disp_field / 2) * (torch.tensor([H, W], device=grid.device) - 1)).flip(-1)
    disp_field = disp_field.permute(1, 2, 0, 3).detach().cpu().numpy()

    return disp_field

Let us know if you have any further questions etc.

Best wishes, Lasse

 Last edited by: lassehansen on July 20, 2024, 10:43 p.m., edited 1 time in total.

Re: Scripts for converting pytorch grid into L2R'24 COMULISSHGBF Submission Format?  

  By: huboqiang on July 21, 2024, 10:16 a.m.

Great, this is exactly what I need.