Add huggingface llava

[jp1924]

Summary

#514
transformer

pip install git+https://github.com/huggingface/transformers.git@5fa35344755d8d9c29610b57d175efd03776ae9e

Testing Done

huggingface-env

• transformers: 4.49.0.dev0
• huggingface_hub version: 0.26.2
• Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• Running in iPython ?: No
• Running in notebook ?: No
• Running in Google Colab ?: No
• Running in Google Colab Enterprise ?: No
• Token path ?: /root/.cache/huggingface/token
• Has saved token ?: True
• Who am I ?: jp1924
• Configured git credential helpers:
• FastAI: N/A
• Tensorflow: N/A
• Torch: 2.5.1+cu121
• Jinja2: 3.1.4
• Graphviz: N/A
• keras: N/A
• Pydot: N/A
• Pillow: 11.0.0
• hf_transfer: N/A
• gradio: N/A
• tensorboard: N/A
• numpy: 2.1.3
• pydantic: 2.9.2
• aiohttp: 3.11.6
• ENDPOINT: https://huggingface.co
• HF_HUB_CACHE: /root/.cache/huggingface/hub
• HF_ASSETS_CACHE: /root/.cache/huggingface/assets
• HF_TOKEN_PATH: /root/.cache/huggingface/token
• HF_STORED_TOKENS_PATH: /root/.cache/huggingface/stored_tokens
• HF_HUB_OFFLINE: False
• HF_HUB_DISABLE_TELEMETRY: False
• HF_HUB_DISABLE_PROGRESS_BARS: None
• HF_HUB_DISABLE_SYMLINKS_WARNING: False
• HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
• HF_HUB_DISABLE_IMPLICIT_TOKEN: False
• HF_HUB_ENABLE_HF_TRANSFER: False
• HF_HUB_ETAG_TIMEOUT: 10
• HF_HUB_DOWNLOAD_TIMEOUT: 10

torch&hw-env

Collecting environment information...
PyTorch version: 2.5.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe

Nvidia driver version: 560.35.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz
CPU family: 6
Model: 106
Thread(s) per core: 2
Core(s) per socket: 12
Socket(s): 2
Stepping: 6
CPU max MHz: 3300.0000
CPU min MHz: 800.0000
BogoMIPS: 4200.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 1.1 MiB (24 instances)
L1i cache: 768 KiB (24 instances)
L2 cache: 30 MiB (24 instances)
L3 cache: 36 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-11,24-35
NUMA node1 CPU(s): 12-23,36-47
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] numpy==2.1.3
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] torch==2.5.1+cu121
[pip3] torchvision==0.20.1+cu121
[pip3] triton==3.1.0
[conda] Could not collect

• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[jp1924]

make test-convergence log

HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/test_mini_models.py
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /root/Liger-Kernel
configfile: pyproject.toml
collecting ...
--------------------------------------------------------------------------------------------------------------------- live log collection ----------------------------------------------------------------------------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 19 items

test/convergence/test_mini_models.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] PASSED                                                                                                                          [  5%]
test/convergence/test_mini_models.py::test_mini_model[mini_llama3-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                 [ 10%]
test/convergence/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                            [ 15%]
test/convergence/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                  [ 21%]
test/convergence/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype4-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                           [ 26%]
test/convergence/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype5-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                 [ 31%]
test/convergence/test_mini_models.py::test_mini_model[mini_qwen2-32-0.0001-dtype6-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                            [ 36%]
test/convergence/test_mini_models.py::test_mini_model[mini_qwen2-32-0.0001-dtype7-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                  [ 42%]
test/convergence/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype8-1e-05-0.1-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                           [ 47%]
test/convergence/test_mini_models.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype9-0.001-0.05-0.1-0.01-0.01-0.01] PASSED                                                                                                                               [ 52%]
test/convergence/test_mini_models.py::test_mini_model[mini_phi3-32-0.0001-dtype10-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                            [ 57%]
test/convergence/test_mini_models.py::test_mini_model[mini_phi3-32-0.0001-dtype11-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                  [ 63%]
test/convergence/test_mini_models.py::test_mini_model[mini_mistral-32-0.0001-dtype12-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                         [ 68%]
test/convergence/test_mini_models.py::test_mini_model[mini_mistral-32-0.0001-dtype13-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                               [ 73%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma1-32-0.0001-dtype14-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                         [ 78%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma1-32-0.0001-dtype15-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                                [ 84%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype16-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                       [ 89%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype17-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                              [ 94%]
test/convergence/test_mini_models.py::test_mini_model[mini_gemma2-32-0.0001-dtype18-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                         [100%]

========================================================================================================== 19 passed, 2 warnings in 282.00s (0:04:41) ==========================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/test_mini_models_multimodal.py
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /root/Liger-Kernel
configfile: pyproject.toml
collecting ...
--------------------------------------------------------------------------------------------------------------------- live log collection ----------------------------------------------------------------------------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 6 items

test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype0-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                   [ 16%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen2_vl-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                         [ 33%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                     [ 50%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                           [ 66%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_llava-32-0.0001-dtype4-1e-08-1e-05-0.005-1e-05-0.005-1e-05]
------------------------------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------------------------------
WARNING  liger_kernel.transformers.monkey_patch:monkey_patch.py:166 Support for transformers versions < 4.46.1 will soon be discontinued due to issues with incorrect gradient accumulation.
 Please consider upgrading to avoid potential issues. See details: <https://github.com/huggingface/transformers/pull/34191>
PASSED                                                                                                                                                                                                                                                   [ 83%]
test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_llava-32-0.0001-dtype5-0.001-0.01-0.1-0.01-0.01-0.01]
------------------------------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------------------------------
WARNING  liger_kernel.transformers.monkey_patch:monkey_patch.py:166 Support for transformers versions < 4.46.1 will soon be discontinued due to issues with incorrect gradient accumulation.
 Please consider upgrading to avoid potential issues. See details: <https://github.com/huggingface/transformers/pull/34191>
PASSED                                                                                                                                                                                                                                                   [100%]

========================================================================================================== 6 passed, 15 warnings in 216.30s (0:03:36) ==========================================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence/test_mini_models_with_logits.py
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /root/Liger-Kernel
configfile: pyproject.toml
collecting ...
--------------------------------------------------------------------------------------------------------------------- live log collection ----------------------------------------------------------------------------------------------------------------------
INFO     datasets:config.py:54 PyTorch version 2.5.1+cu121 available.
collected 19 items

test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] PASSED                                                                                                              [  5%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype1-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                     [ 10%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llava-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05]
------------------------------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------------------------------
WARNING  liger_kernel.transformers.monkey_patch:monkey_patch.py:166 Support for transformers versions < 4.46.1 will soon be discontinued due to issues with incorrect gradient accumulation.
 Please consider upgrading to avoid potential issues. See details: <https://github.com/huggingface/transformers/pull/34191>
PASSED                                                                                                                                                                                                                                                   [ 15%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llava-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01]
------------------------------------------------------------------------------------------------------------------------ live log call -------------------------------------------------------------------------------------------------------------------------
WARNING  liger_kernel.transformers.monkey_patch:monkey_patch.py:166 Support for transformers versions < 4.46.1 will soon be discontinued due to issues with incorrect gradient accumulation.
 Please consider upgrading to avoid potential issues. See details: <https://github.com/huggingface/transformers/pull/34191>
PASSED                                                                                                                                                                                                                                                   [ 21%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_mllama-32-0.0001-dtype4-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                               [ 26%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_mllama-32-0.0001-dtype5-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                     [ 31%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_qwen2-32-0.0001-dtype6-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                [ 36%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_qwen2-32-0.0001-dtype7-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                      [ 42%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype8-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [ 47%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_qwen2_vl-32-0.0001-dtype9-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                   [ 52%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_phi3-32-0.0001-dtype10-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                                [ 57%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_phi3-32-0.0001-dtype11-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                      [ 63%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_mistral-32-0.0001-dtype12-1e-08-1e-05-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [ 68%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_mistral-32-0.0001-dtype13-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                   [ 73%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma1-32-0.0001-dtype14-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [ 78%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma1-32-0.0001-dtype15-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                    [ 84%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype16-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                           [ 89%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma1.1-32-0.0001-dtype17-0.001-0.01-0.1-0.01-0.01-0.01] PASSED                                                                                                                  [ 94%]
test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_gemma2-32-0.0001-dtype18-1e-08-0.0001-0.005-1e-05-0.005-1e-05] PASSED                                                                                                             [100%]

========================================================================================================== 19 passed, 2 warnings in 258.24s (0:04:18) ==========================================================================================================

[jp1924]

cc @Tcc0403 @ByronHsu

[Tcc0403]

Good work! I'll take a look in few days.

[Tcc0403]

There are multiple breaking changes in transformers recently. Convergence test couldn't pass with newer transformers version.

    def test_mini_model(
        model_name,
        num_steps,
        lr,
        dtype,
        loss_atol,
        loss_rtol,
        logits_atol,
        logits_rtol,
        param_atol,
        param_rtol,
    ):
        # Non-liger models should be initialized and tested first to avoid the module being overridden

>       expected_output = run_mini_model(model_name=model_name, num_steps=num_steps, dtype=dtype, lr=lr)

test/convergence/test_mini_models.py:773:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test/convergence/test_mini_models.py:519: in run_mini_model
    output = model(**batch)
.venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1736: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
.venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1747: in _call_impl
    return forward_call(*args, **kwargs)
.venv/lib/python3.10/site-packages/transformers/models/llava/modeling_llava.py:492: in forward
    inputs_embeds, attention_mask, labels, position_ids = self._merge_input_ids_with_image_features(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = LlavaForConditionalGeneration(
  (vision_tower): CLIPVisionModel(
    (vision_model): CLIPVisionTransformer(
      (em...rotary_emb): LlamaRotaryEmbedding()
    )
    (lm_head): Linear(in_features=1024, out_features=32064, bias=False)
  )
)
image_features = None
inputs_embeds = tensor([[[ 1.1444e-05, -1.3367e-02,  4.3640e-03,  ...,  1.9043e-02,
           3.6377e-02, -8.7891e-03],
         [ 1....-02,
           5.7373e-03, -9.4604e-04]]], device='cuda:0', dtype=torch.bfloat16,
       grad_fn=<EmbeddingBackward0>)
input_ids = tensor([[    2,     2,     2,  ..., 16334, 20084, 28747],
        [    2,     2,     2,  ...,   528,  4085, 28723],
  ...,     2,     2,  ...,     2,     2,     1],
        [    2,     2,     2,  ...,     1,  1682, 28747]], device='cuda:0')
attention_mask = tensor([[0, 0, 0,  ..., 1, 1, 1],
        [0, 0, 0,  ..., 1, 1, 1],
        [0, 0, 0,  ..., 0, 0, 1],
        ...,
        [0, 0, 0,  ..., 1, 1, 1],
        [0, 0, 0,  ..., 0, 0, 1],
        [0, 0, 0,  ..., 1, 1, 1]], device='cuda:0')
labels = tensor([[    2,     2,     2,  ..., 16334, 20084, 28747],
        [    2,     2,     2,  ...,   528,  4085, 28723],
  ...,     2,     2,  ...,     2,     2,     1],
        [    2,     2,     2,  ...,     1,  1682, 28747]], device='cuda:0')

    def _merge_input_ids_with_image_features(self, image_features, inputs_embeds, input_ids, attention_mask, labels):
>       num_images, num_image_patches, embed_dim = image_features.shape
E       AttributeError: 'NoneType' object has no attribute 'shape'

.venv/lib/python3.10/site-packages/transformers/models/llava/modeling_llava.py:303: AttributeError
------------------------------------------------- Captured stdout call -------------------------------------------------
Liger kernel patches have been reverted.
=============================================== short test summary info ================================================
FAILED test/convergence/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] - AttributeError: 'NoneType' object has no attribute 'shape'
FAILED test/convergence/test_mini_models.py::test_mini_model[mini_llava-32-0.0001-dtype3-0.001-0.01-0.1-0.01-0.01-0.01] - AttributeError: 'NoneType' object has no attribute 'shape'

Environment:

❯ python -m liger_kernel.env_report
Environment Report:
-------------------
Operating System: Linux-5.15.133.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
Python version: 3.10.12
Liger Kernel version: 0.5.2
PyTorch version: 2.5.1+cu124
CUDA version: 12.4
HIP(ROCm) version: Not available
Triton version: 3.1.0
Transformers version: 4.47.1
XPU version: XPU Not Available

Let's make a condition to handle different function signatures for older and newer transformers version, since both are still being used by users.

something like

if transformer_version < ...:
  ...
else:
  ...

[jp1924]

@Tcc0403
Ah, this is an error from transformers
You can see the exact cause in this link: huggingface/transformers#35526
However, this issue has been fixed in the latest version of transformers on GitHub, so to run it correctly, you will need to install from the source using pip in the transformers repo.

pip install git+https://github.com/huggingface/transformers.git@5fa35344755d8d9c29610b57d175efd03776ae9e

[jp1924]

That's why if you look at the hw&sw specification, my transformer is 4.49.0.dev instead of 4.48.0 due to this issue.

[jp1924]

make test result

================================== 854 passed, 215 skipped, 44 warnings in 427.70s (0:07:07) ==================================

[Tcc0403]

Resolve conflicts then we can merge it after I run the test again!

[jp1924]

@Tcc0403
The conflict occurred because #498 was merged without applying the code style separately.
So, there's nothing I can do on my end right now.
To resolve this, we either need to open a separate PR or directly fix the code.

[Tcc0403]

We can just resolve it in this PR by applying the correct style.

[jp1924]

@Tcc0403
I have resolved the conflicts for now. Please check it.

[jp1924]

We can just resolve it in this PR by applying the correct style.

Since my PR already has the codestyle applied,
I couldn't resolve the conflicts using any method. Therefore, I resolved them based on #498.

[jp1924]

test_mini_models.py

============================================================================================ 19 passed, 1 warning in 184.30s (0:03:04) ============================================================================================

test_mini_models_with_logits.py

===================================================================================================== short test summary info =====================================================================================================
FAILED ../../../../home/jp/workspace/Liger-Kernel/test/convergence/test_mini_models_with_logits.py::test_mini_model[mini_llama3-32-0.0001-dtype0-1e-08-2e-05-0.0001-1e-05-0.005-1e-05] - AssertionError: Number of mismatched elements: 65530177
======================================================================================= 1 failed, 18 passed, 1 warning in 175.00s (0:02:54) =======================================================================================

test_mini_models_multimodal.py

===================================================================================================== short test summary info =====================================================================================================
FAILED ../../../../home/jp/workspace/Liger-Kernel/test/convergence/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] - AssertionError: Number of mismatched elements: 65236533
======================================================================================= 1 failed, 5 passed, 7 warnings in 159.10s (0:02:39) =======================================================================================

@Tcc0403
Llava passes the tests, but Llama does not.
I tested both the add_llava branch and the latest main branch, and the results are the same.

[Tcc0403]

I'm not sure whether it is related to these failures. But currently there seems to be some bugs in revert functions for convergence test. It might be another compatibility issues with transformers version. I'll investigate it soon.

[Tcc0403]

After some investigations, the convergence tests regarding comparison of logits would fail with transformers>=8ebe9d7 because of the mixture of #542 and #543.

I have to handle these issues asap before getting back to this PR to ensure the correctness of ci.

[jp1924]

Ah, understood.
I'll stop working until that issue is resolved.
Simultaneously, I'll also take a look to help resolve the issue.

[jp1924]

@Tcc0403

  modal run dev.modal.tests
  shell: /usr/bin/bash -e {0}
  env:
    MODAL_TOKEN_ID: 
    MODAL_TOKEN_SECRET: 
    pythonLocation: /opt/hostedtoolcache/Python/3.10.16/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.16/x64/lib
╭─ Modal Deprecation Warning (2024-01-08) ─────────────────────────────────────╮
│ modal.Mount usage will soon be deprecated.                                   │
│                                                                              │
│ Use image.add_local_dir instead, which is functionally and performance-wise  │
│ equivalent.                                                                  │
│                                                                              │
│ Source: /home/runner/work/Liger-Kernel/Liger-Kernel/dev/modal/tests.py:14    │
│   repo = modal.Mount.from_local_dir(ROOT_PATH, remote_path=REMOTE_ROOT_PATH) │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Token missing. Could not authenticate client. If you have token credentials, │
│ see modal.com/docs/reference/modal.config for setup help. If you are a new   │
│ user, register an account at modal.com, then run `modal token new`.          │
╰──────────────────────────────────────────────────────────────────────────────╯

I keep getting emails with the subject [jp1924/Liger-Kernel] NVIDIA GPU workflow run. It seems like it keeps failing because of a missing authentication token. Does anyone know how to fix this?