Veteran editor / graphics guy familiar with 3-D has question about AI Studio workflow

I love the concept of what Iclone does to take back control from the models. Today was my maiden flight. Unfortunately it crashed and burned and I’m hoping to find out why and what options I have. I created a 10 second animation of a camera flying in toward a haunted house with a character standing by the front door. Very simple. And I’ve rendered it out for use in AI studio. So it had all the appropriate textures from the various characters and props I bought from re-illusion. I followed the process of having the video imported into AI Studio which was impressive and then it asked me to also laod the first frame of the video be a stylistic or representational look reference for the video. And this is where it fell apart the first time. What it did was import the first frame from my low resolution intentionally made this way video to be used merely as a guide for motion and timing knowing full well that AI studio would integrate the high resolution graphics over the top. In hindsight a friend suggested I should’ve grabbed the first frame and put it in nano banana 2 in order to create a photo realistic high resolution image. But I did not see any way of uploading that altered image which really would show the style I wanted back into that little box that said basically first frame here. So when I generated the animation the first thing I noticed is how crummy it looked . it looked like a low rez animation. And then I realized of course it did. It had imported the first frame from an animation only intended to be a guide for the controllednet process. Is it possible to get a altered frame into that box which basically says this will determine the stylistic look of your video? I really hope that is the case cause otherwise this whole process makes no sense to me whatsoever. And to save money it was suggested I use not the various high-end model models I’m used to but one that started with the letter S and had a funny name sorry I don’t remember. To my dismay not only did the shot look Lores and we know why now ,but halfway through the 10 second shot which really was just a moving camera up a walkway to somebody standing by a door , the model decided to cut the animation and rerun the opening so I had five seconds of the open followed by five seconds of the open. II am hoping that’s just a result of using a lesser model. This hybrid approach seems to be a wonderful solution to getting temporal and spatial control away from the models and let them do the imagery and wrap it around the guide animation. That’s what it was designed for. But I haven’t seen anybody online saying they’ve used this process and it’s making me quite nervous. I really hope somebody can help inform me how to correct these two really bad things happening. Thanks!

1 Like

I figured it out soon after this massive post was made. But I’m new so it took several days before I got approval of the final posting. Here’s what I figured out. After creating the opening animation for my next project I rendered the animation out with basic textures and a full set with characters. Within the 10 second scene there was a massive one shot camera move as well as a minor move by one of the primary characters. When I output this animation I then uploaded it to AI studio which uses software called control net to take my fully textured video and strip off the texture and replace it with depth mapping. This allows the most effective and efficient overlay or wrapping of AI created imagery over my basic animation. And I can add any enhancements by using a prompt simultaneously as well. My first problem was after AI studio requested I import my full animation to its viewport it then requested I use the opening frame as a stylistic reference for the rest of the animation. And I just fed it the first frame of this low quality video which in hindsight I realized was my first mistake. When I figured out what was not working I did a screen grab of the first frame brought it into nano banana too and prompted something simple like haunted house cinematic photo realistic. I then took that picture and use that as the single frame style reference that is necessary for this process. After generating that animation I was much more pleased with the results and saw lots of potential. My second mistake was trying to save money by using one of the lesser models that I’ve never heard of but others have promoted as a good cheap alternative and unfortunately the model only rendered half of my 10 second animation and then repeated the first five seconds again. But when I stepped up to the plate and paid for C dance 2.0 everything looked and animated just as I intended. This was quite a relief and a joyous experience because I love the process of creating that one shot open which starts out as a elevated tilted down view of the house and then flies down through the gate up to the front door. It just looks really good but I see room for improvement in the area of realism. And I will work with the LLM’s to add that enhancement which doesn’t quite hit the mark. But being able to lock the camera move into constant repeatability and always hitting its key frames at the same time was wonderful just as having the characters repeatedly move the way I wanted and hitting their marks and delivering their lines on time is such a treat. So much better than pray and pay. Thank you re-illusion.

I sat down to write a step by step workflow for making a video. All gleaned from the manual and video tutorials. It helped sort things out in my confused mind . :roll_eyes: Not the last word on the subject but maybe it helps you too.


Creating Videos - recommended to use Seedance
Mike’s Intro
This part is for creating multi shots in Iclone. Skip if only one shot to render.

1] Add Character/s to scene in iclone (Note greyed characters helps get better AI render results)

2] With Preview Camera - position/pose the character/s[or animate ]
3] Add Camera 1 - position camera ie for your first shot
4] Add Camera 2 - " " and focus as required for next shot
5] Add more Cameras for further shots if required.
6] when done select ‘Switch Camera’ from Cam menu
A switcher is added to the time line.
7] On the SWITCH CAM TIME LINE select the FIRST frame and right click to apply first camera key frame.
8] move cursor to where you want next shot. Right click and select Camera 2 to set a key frame
9] Repeat for all cameras.

10] drag cursor to see camera switches are flowing as desired.

IMPORTANT
11] Put cursor on 1st Frame and click ICON 'capture image"(next to AI render Tab)
This will send a reference image to AI Studio (opens Automatically )
12]In AI Studio in"2D TO IMAGE"Add a detailed PROMPT EG-location,environment etc

13] Using styles icon choose render style.
14] GENERATE image.

15] Switch to “3D to VIDEO”
Select render model, Video ratio & Resolution, Seedance preferred for best reults.
Select “Capture Video from 3D viewport”.
Select “'Apply camera movement”
nothing in the ref 1 box

16] Next in the prompt, describe the “action” you want to happen in the scene
and include any dialogue.
17] Select RENDER.

USING GREY CHARACTERS

Same procedure as above but before ‘rendering’ you must add an AI ACTOR.

18] right click on the ‘reference box’ select either character or audio.
select Actor to open library. Click on required ACTOR which will transfer to reference showing its prompts[ editable ]
ADD more actors as necessary.

USING DUMMY CHARACTER from content [ in lieu of greyed textures ]

19] Repeat all above steps - The difference is, dummies have no facials and allow renderer to manage expressions based on the prompts and don’t compete with facial animation.

However Dummies need to be coloured and the colours refenced in prompts so that AI know which object to apply the prompt to.

20] In 3D to Image, after generating the first frame (Step 11 above) Apply Actor/s from library
[ can be one you’ve created your self ]
and then ASSIGN. [ opens window where you apply a grid over the Actors head.] Click Apply.

21] Add prompt describing action making sure you specify which dummy is which. eg Pink dummy character is CAMILA etc.