Kwai-Kolors/Kolors-Inpainting · Problem with using inpainting model with ip-adapter

is this kind of connecting these pipelines is impossible because of unet differences? i always get mismatch of mat here :

inpainting_pipeline.load_ip_adapter(
f"{root_dir}/Kolors/weights/Kolors-IP-Adapter-Plus",
subfolder="",
weight_name="ip_adapter_plus_general.bin",
low_cpu_mem_usage=True
)

result = inpainting_pipeline(
    prompt=prompt,
    image=image,
    mask_image=mask_image,
    height=1024,
    width=1024,
    guidance_scale=6.0,
    generator=generator,
    num_inference_steps=25,
    ip_adapter_image=ip_image,
    negative_prompt='deformed hands, low quality, blur, distortion',
    num_images_per_prompt=1,
    strength=0.999
).images[0]