Self-hosting Moondream

​

Running Moondream locally

To run Moondream on your local machine, first go to https://moondream.ai/c/moondream-server and download the appropriate server (currently Moondream only provides macOS/Linux executables).

Follow the instructions on that page to run the executable.

Then you can configure magnitude.config.ts with the local base URL:

import { type MagnitudeConfig } from 'magnitude-test';

export default {
    url: "http://localhost:5173",
    executor: {
        provider: 'moondream',
        options: {
            baseUrl: 'http://localhost:2020/v1'
        }
    }
} satisfies MagnitudeConfig;

That’s it! Now you can run tests with local Moondream.

​

Hosting in a VPC

If your company has strict requirements around where infrastructure needs to be hosted, you may need to host Moondream inside your company’s VPC on your cloud provider of choice.

The specific setup required will depend on the cloud provider.

However generally speaking these are the requirements:

1. A compute instance in your VPC with a GPU
2. A linux distribution with CUDA configured on the compute instance
3. Run the Moondream server executable on this instance
4. Ensure that the instance is accessible via port 2020 to your CI runners or dev machines, and configure test runners to connect to the appropriate URL:

import { type MagnitudeConfig } from 'magnitude-test';

export default {
    url: "http://localhost:5173",
    executor: {
        provider: 'moondream',
        options: {
            baseUrl: 'https://some-host-in-my-vpc/v1'
        }
    }
} satisfies MagnitudeConfig;

If you are an enterprise trying to configure Magnitude in your company, feel free to reach out to us directly at founders@magnitude.run.

​

Self-hosting Moondream on Modal

Probably the easiest way to deploy Moondream is by hosting it on Modal, so we provide a deployment script to do this.

​

Set up Modal

1. Create an account at modal.com
2. Run pip install modal to install the modal Python package
3. Run modal setup to authenticate (if this doesn’t work, try python -m modal setup)

More info: https://modal.com/docs/guide

​

Deploy Moondream

Clone Magnitude and run the moondream.py deploy script to automatically download the Moondream server and model weights to a Modal volume and deploy the endpoint:

git clone https://github.com/magnitudedev/magnitude.git
cd magnitude/infra
modal deploy moondream.py

Your Moondream endpoint will then be available at https://<your-modal-username>--moondream.modal.run/v1.

Then you can configure magnitude.config.ts with this base URL and start running tests powered by your newly deployed Moondream server!

import { type MagnitudeConfig } from 'magnitude-test';

export default {
    url: "http://localhost:5173",
    executor: {
        provider: 'moondream',
        options: {
            baseUrl: 'https://<your-modal-username>--moondream.modal.run/v1'
        }
    }
} satisfies MagnitudeConfig;

Keep in mind there may be some cold start times since Modal automatically scales down containers when not in use.

​

Customizing Deployment

You may want to customize the modal deployment depending on your needs.

You can modify the moondream.py deployment script to fit your needs.

Common options you may want to change:

• gpu: GPU configuration. See Modal’s pricing page for details on different available GPUs and their cost. Also see comparison below.
• scaledown_window: Time in seconds a container will wait to shut down after receiving no requests. If higher, will let you run tests after longer without needing the container to cold-start again.
• min_containers: By setting this option you force Modal to keep some number of containers open to handle requests. This means Modal will also bill you for those containers all the time, but eliminates cold-starts.

​

GPU Comparison

Here’s a breakdown of how quickly different GPUs available on Modal are able to handle typical requests that Magnitude makes to Moondream:

Since Magnitude needs to wait a bit for the page to stabilize anyway, probably something like the A10G is a good price/performance balance, but any of these work well!