Guidance on Background Removal Task with 10K Annotated Car Images – Fine-Tuning vs Training from Scratch

[alichohann]

Hello,

First of all, I want to express my gratitude for the amazing work done on Bi-RefNet – it’s an impressive model! I’m currently working on a use case where I need to remove backgrounds from a set of images. I have around 10K images, each with varying resolutions (up to 4K). These images contain cars, and I’ve annotated them at the pixel level, segmenting individual cars, with some images featuring one car and others containing multiple cars.

I would like to ask for your expert opinion and recommendations on the best approach for my use case:

1. Should I fine-tune the model on my dataset or train it from scratch?
   • Given that I have a decent-sized dataset with 10K images and pixel-level annotations, I’m wondering whether fine-tuning an existing model would be more efficient than starting from scratch.
2. What weights should I use for training?
   • If fine-tuning is the recommended path, I would appreciate your suggestion on which pre-trained weights I should use.
   • If training from scratch is more suitable, could you provide any recommendations on the model configuration, loss functions, or other considerations specific to my task of background removal for segmented car images?

I would be grateful for any guidance or pointers on how to best approach training Bi-RefNet for my use case.

Thank you in advance for your help!

[ZhengPeng7]

Hi, thanks for the appreciation.
My personal views on your questions:

1. Train from scratch only when you have enough data to approach the very great performance without any worry of cost. Otherwise, such as the existing models can already make acceptable predictions, the training cases are not very special ones, etc., I suggest finetuning.
2. You can try those models to see which one can do the best zero-shot prediction on your cases. Choosing this should be mostly right.

You're welcome. I'm not sure about how the models perform on those cases with multiple cars. If they can only identify the most salient foreground one, you can also simultaneously train from scratch on only the images containing multiple cars, with a little resource, to see if it can learn this clue.