can't make it run and read a webcam on Ubuntu 24.04

[4aiman]

Tried installing from local and from pypi.
Always fail with:

cv2.error: OpenCV(4.10.0) /io/opencv/modules/highgui/src/window.cpp:1367: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvWaitKey'

when running like this

transparent-background -s 2 -d 4 --type green --mode fast

I do have both libgtk2.0-dev (2.24.33-4ubuntu1.1) and pkg-config (1.8.1-2build1).
Still, I'm not sure how I can cmake anything, since there's no CMakeLists.txt anywhere in this repo.

transparent-background-gui

does launch but can open a webcam but has conflicting info on what's being used: map_location is on CPU but the device is set to cuda:0 (why?)

In the end it's unusable

[plemeri]

Hi @4aimanl, did you install with below instruction?
https://github.com/plemeri/transparent-background?tab=readme-ov-file#with-webcam-support-not-stable

Also, webcam input is not tested for a long time, so it might not working after all. If you still need to process webcam input, you can try building your own program by using our package as a python API. Feel free to ask more.

[4aiman]

Yeah, tried that as well.

In fact, even if I omit the webcam, as soon as I try to use it, python demands the module and stirs me towards installing it.

With the API method I have several questions.

Hope you have time and willingness to answer at least some of them 😅

Here's the code I'm testing:

import cv2
import numpy as np
import pygame

from PIL import Image
from transparent_background import Remover
width, height = 320, 240
screen = pygame.display.set_mode((width, height))
cap = cv2.VideoCapture(2)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

# Load model
remover = Remover(jit=True, device='cuda:0', ckpt='ckpt_fast.pth', mode='fast') # nightly release checkpoint

def get_frame():
    ret, frame = cap.read()
    if not ret:
        return None
    return frame


running = True
while running:
    frame = get_frame()
    if frame is not None:
        out = remover.process(frame, type="[0,255,0]", threshold=0.6, reverse=False)
        out = cv2.cvtColor(out, cv2.COLOR_BGR2RGB)
        frame = pygame.surfarray.make_surface(out)
        frame = pygame.transform.rotate(frame, -90)
        
        screen.blit(frame, (0,0))
        pygame.display.update()

cap.release()
cv2.destroyAllWindows()

1. Is there a way to use pygame's images instead of cv2 ones inside remover.process?
   I think (quite ignorantly, I'm doing python for about a week) that would save me a lot of overhead since I have an entire interface built around pygame I'd like to reuse in the future.

2. Is there a way to speed up the processing?
   It seems that too rapid movements confuse the processor and it can't "see" hands (it's enough to "wave" into camera to see the effect)

3. Is there a way to lower hardware load?
   I did try lowering the resolution, of course. But regardless of it being 640x480 or 320x240, I'm still using 40% of RTX4070.

4. There's an error in the example on how to change the background color.

   1. First of all, what's supposed to be "green" is more like a pastel lemongrass(?)
      

   2. Secondly, specifying (as per example) type=[0,255,0] in the options for remover.process yields an error:

   ...site-packages/transparent_background/Remover.py", line 200, in process
       if type.startswith("["):
          ^^^^^^^^^^^^^^^
   AttributeError: 'list' object has no attribute 'startswith'
   

   This can be resolved with supplying a string instead of an array like so: type="[0,255,0]".

5. Can I suppress the output?
   It's fine to get some initial info (not sure I understand all of it), but spamming the BG color every frame is excessive to say the least.

./venv/bin/python test11.py
pygame-ce 2.5.2 (SDL 2.30.8, Python 3.12.3)
/home/4aiman/dev/mm/venv/lib/python3.12/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 1.4.24 (you have 1.4.15). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
  check_for_updates()
/home/4aiman/dev/mm/venv/lib/python3.12/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/4aiman/dev/mm/venv/lib/python3.12/site-packages/torch/functional.py:534: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3595.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/4aiman/dev/mm/venv/lib/python3.12/site-packages/transparent_background/Remover.py:92: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  torch.load(os.path.join(ckpt_dir, ckpt_name), map_location="cpu"),
Settings -> Mode=fast, Device=cuda:0, Torchscript=enabled
[0, 255, 0]
[0, 255, 0]
[0, 255, 0]
[0, 255, 0]
[0, 255, 0]
[0, 255, 0]
[0, 255, 0]
[0, 255, 0]
...

6. I can't get dynamic resize to work.
   Online sources recommend uninstalling headless cv, but that leads to a whole barrage of other errors