ORPO on SFT dataset

[vitalyshalumov]

System Info

Hello,
As I understand it ORPO leverages SFT and preference in one step.
But what if I have only SFT data without prefenrece?
Can I still use ORPO with unpaired_preference?
Or the way to go is SFT and then ORPO on a prefenrece dataset if I have one?

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

from trl import ...

outputs:

Traceback (most recent call last):
  File "example.py", line 42, in <module>
    ...

Expected behavior

How to leverage ORPO

Checklist

• I have checked that my issue isn't already filed (see open issues)
• I have included my system information
• Any code provided is minimal, complete, and reproducible (more on MREs)
• Any code provided is properly formatted in code blocks, (no screenshot, more on code blocks)
• Any traceback provided is complete