ControlNet and Reference Images: Precise Control Over AI Video

Go beyond text prompts. Learn how to use ControlNet, pose references, depth maps, and image conditioning to get exactly the composition and motion you want.

Beyond Text Prompts

Text prompts are powerful, but they have limits. If you want precise control over composition, pose, camera angle, or motion, you need more advanced techniques. That is where ControlNet and reference images come in.

This tutorial covers the advanced methods the pros use to get consistent, controllable results from AI video.

What is ControlNet?

ControlNet is a neural network architecture that adds extra conditioning to AI image and video models. Instead of generating purely from text, ControlNet lets you guide the generation with additional information like:

• Poses and skeletons
• Depth maps
• Edge detection / line art
• Segmentation masks
• Normal maps

Think of it like giving the AI a "blueprint" to follow. The text prompt describes what things look like, and ControlNet describes where things go.

Types of Control

Pose Control (OpenPose)

Uses a skeleton pose to control character positioning and body posture.

• **Best for:** Character animations, consistent poses, specific body language
• **How to get it:** Use an OpenPose detector on a reference image, or manually pose a 3D skeleton

Depth Control

Uses a depth map to control the 3D structure and composition of the scene.

• **Best for:** Scene composition, camera movement, spatial relationships
• **How to get it:** Generate a depth map from any image using depth estimation tools

Canny / Edge Control

Uses edge detection to control the shapes and outlines of objects.

• **Best for:** Character design consistency, architectural accuracy, product shots
• **How to get it:** Run edge detection on a reference image or sketch

Segmentation Control

Uses a color-coded mask to control what objects go where.

• **Best for:** Complex scenes with multiple objects, precise layout control
• **How to get it:** Create a segmentation map manually or with an image segmentation tool

How to Use Reference Images

Reference images are the easiest way to get more control. Most AI video tools support them to some degree.

Image-to-Video

The simplest form of reference-based control: 1. Upload an image 2. Add a text prompt (optional) 3. The AI animates the image 4. You get a video that preserves the composition and subject

This is great for:

• Consistent character appearance
• Precise framing and composition
• Bringing concept art to life
• Starting from a known good composition

Style References

Some tools let you reference the style of an image without copying the composition:

• Upload a style reference image
• The AI matches the color palette, lighting, and art style
• But generates a completely new composition

Great for brand consistency and developing a signature look.

IP-Adapter / Character Reference

Advanced tools support character consistency through specialized reference techniques:

• Upload a reference photo of a character
• The AI keeps that character consistent across generations
• Works across different poses, angles, and scenes
• Essential for any project with recurring characters

Workflow: Character Consistency

One of the biggest challenges in AI video is keeping characters consistent across multiple clips. Here is a professional workflow:

Step 1: Create a Character Sheet

Generate or create several reference images of your character:

• Front view
• 3/4 view
• Side profile
• Different expressions
• Full body and close-up

The more reference angles you have, the better consistency you will get.

Step 2: Use Image-to-Video

Start with your best reference image: 1. Upload the reference 2. Add motion with a text prompt 3. Generate a short clip 4. Check if the character stays consistent

Step 3: Chain Generations

For longer or more complex scenes: 1. Generate the first clip from a reference image 2. Take the best frame from that clip 3. Use that frame as the reference for the next clip 4. Repeat as needed

This "frame chaining" technique maintains consistency much better than generating each clip independently.

Step 4: Use IP-Adapter If Available

If your tool supports IP-Adapter or similar character reference technology:

• Upload 1-3 reference images
• Set the reference strength (0.5-0.8 usually works well)
• Generate with your prompt
• The character should stay recognizable across generations

Workflow: Precise Camera Movement

Controlling camera movement precisely is another area where references help.

Method 1: Video-to-Video

If you have a reference video with the camera movement you want: 1. Upload the reference video 2. Apply style transfer or re-rendering 3. The motion and timing stay the same, but the visuals change 4. You get precise, human-designed camera work with AI visuals

This is a pro trick — film a simple reference (even with your phone), then AI-re-render it.

Method 2: Keyframe Animation

Tools with keyframe control (like Runway Director Mode) let you:

• Set camera position at frame 0
• Set camera position at frame N
• The tool smoothly interpolates between them
• You get exactly the camera movement you designed

Method 3: Depth Map Control

For tools that support depth conditioning: 1. Create or generate a depth map of your scene 2. Use it as control for your video generation 3. The AI respects the 3D structure of the scene 4. Camera movement feels more natural and consistent

Tools with the Best Control

Not all tools offer the same level of control. Here is how they compare:

Runway Gen-4

• **Director Mode** — frame-level camera and character control
• **Image-to-video** — excellent reference image support
• **Video-to-video** — style transfer and re-rendering
• **Best for:** Professional production where control matters

Stable Video Diffusion (open source)

• **Full ControlNet support** — pose, depth, canny, and more
• **Complete control** — tweak every parameter
• **Requires technical knowledge** — not for beginners
• **Best for:** Technical users who want maximum control

Kling 2.0

• **Strong image-to-video** — good reference preservation
• **Motion brush** — control where motion happens
• **Multi-reference** — support for multiple input images
• **Best for:** Balance of control and ease of use

Pika 2.0

• **Image-to-video** — decent reference support
• **Motion control** — adjustable motion strength
• **Easiest to use** — great for beginners
• **Best for:** Getting started with reference-based control

Common Problems and Solutions

Character Changes Between Frames

• Use stronger reference image weight
• Try a different reference angle
• Use shorter clip durations
• Try IP-Adapter or character lock features

Camera Movement Feels Unnatural

• Use video-to-video with real reference footage
• Use keyframe control if available
• Reduce motion strength
• Try simpler camera movements first

Composition Drifts

• Start with a strong reference image
• Use ControlNet (canny or depth) if available
• Keep clip durations shorter
• Generate more variations and pick the best

Practice Exercises

Want to improve your control skills? Try these exercises:

Exercise 1: Pose Consistency

• Find a reference photo of a person in a specific pose
• Generate 5 videos from it
• See how many keep the pose consistent
• Try adjusting the reference strength

Exercise 2: Camera Movements

• Using a tool with camera control, create:
• A slow dolly-in
• A 180-degree orbit
• A pan from left to right
• A tilt from low to high
• Focus on smooth, natural-feeling motion

Exercise 3: Style Transfer

• Take a simple phone video of something
• Use video-to-video to re-render it in different styles
• Try: cyberpunk, watercolor, 3D render, film noir
• See how the motion stays the same but the look changes

Final Thoughts

Mastering control in AI video generation is what separates amateurs from professionals. Anyone can type a prompt and get something cool — but being able to consistently produce exactly what you envision is a valuable skill.

Start simple with image-to-video references, then gradually explore more advanced techniques like ControlNet and keyframe animation. As you gain more control, your AI videos will become more professional, more consistent, and more effective.