Skip to content

any way to pipeline from generators or infinite pipelines? #17

Description

@FarisHijazi

I'm using mpipe for computer vision processing from camera frames, it's exactly what I need, but the put() get() api is nonideal, and I can't figure out a way to have infinite tasks.

I read the documentation but I keep finding that you need to feed the tasks before running, I need to feed tasks while running

stage1.link(stage2)
...

pipe = Pipeline(...

for number in range(10):
    pipe.put(number)

I tried running a thread in the background to feed a multiprocessing.Queue but it just stops when the queue gets full, I concluded that you can't run put() while you're running get()

def fill_queue():
    while True:
        if not q.full():
            pipe.put(0)
        else:
            time.sleep(0.01)  # Rest for 10ms, we have a full queue
thread = Thread(target=fill_queue, args=())
thread.daemon = True


while True:
    if q.qsize():
       results... = q.get()

either way, something is very wrong

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions