Lightly edited from the recording for readability. Wording is the speakers’ own.
Today, welcome to this week's FLORA webinar featuring Ross Symons, Chief Creative Officer from Zenrobot. All right, Ross, over to you. Let's do it. Awesome. Cool. Thank you, Olivia. Yeah.
Hello, everyone. My name is Ross. I am the Chief Creative Officer at Zenrobot. We are a gen AI company based in Cape Town, South Africa. We've been fortunate enough to train. We have two parts of the business. We do AI training for creatives, helping creatives, marketing teams implement AI tools into their creative workflows.
We also do custom training, AI training programs as well for businesses and companies around the world, as well as content production. So yeah, we're a very busy team and it's loads of fun. It's an absolute honor to be here today partnering with FLORA. This is the first of a series of content pieces we will be creating today being the first one, which is taking a single image, how I started with the single image and created a three-minute film, something I've been wanting to do on this scale for quite a while.
So yeah, super excited to get into the details of all of this. So we're going to take a look at the film. I'll go through a deck of a couple of things that just on my process and how it worked. We'll do some Q&A at the end.
But my intention behind this demo is really just to show the capabilities of what you can do within FLORA and also just tying it into film as well. I mean, FLORA, whether you have or haven't used it before, you can do pretty much anything in terms of video, audio and an image. So yeah, very powerful tool, super fun to work with. Yeah, and I hope you learned something today.
So what we're going to do is we're going to take a look at the film itself. We'll watch that and then we'll get into the demo itself. Get lost, freak. Coming up, I loved everything about this character, about how lonely he looks, just this ghost in a skate park. It was very random, but I thought, what better opportunity to use this?
So I thought, let's take this image and yeah, just see what we can do kind of with it. When I'm starting out with a project like this, I do attempt to use the image itself with a large language model, kind of reverse engineering the basic properties of what the image entails. So looking at the light, the color, the environment and composition and breaking it down into a text format. The reason I do that is plainly because using an image as a reference alongside using a prompt just makes the results a lot more consistent and a lot more rich. And this rings true for both image generation as well as when you are creating videos as well.
So I use chat GPT in this case, extracted the components and the composition of this, everything about the character and the environment, the lighting, and then use that as a sort of a guide with all the images going forward. When I created the story itself, I had this idea in my head, spoke to chat, my friend chat. I was just this is what I want to do you have any just thoughts on what the, I guess, the foe of this could be? It pointed out a couple of things, just, where I should introduce the pink or the savior ghost at the end.
But it was just a story about this ghost that goes through this world and he's kind of lonely and someone sticks up for him, or he kind of sees, he's seen by someone else at the end. So chat helped me with this, drew the storyboard up. And I referred back to this when I was kind of laying out the scenes and eventually working with Claude, dropping that into Claude, connecting everything via MCP, which we're going to get into the details of that as well. But this was the starting block of that. I really struggled to get consistency in the first version.
So when I started this project out, I actually did this twice. I made the film. I didn't finish it the first, but the first time I created a canvas, I was working directly in chat GPT, but I found that I wasn't getting consistency. The character was kind of falling apart. There were just little details. Like when the sheet lifted the color of the pants underneath, sometimes his hand showed or his arm showed.
So I just wanted it all to be consistent. So I wasn't getting that consistency, particularly with the environments as well. The skate bowl needed to be the same from every angle. So what I eventually did is I pulled everything into, this is just a render of the sort of gray plate, which I made in, I say I made Claude assisted me in making in blender, which I connected via MCP.
But as soon as I moved to Claude and I started kind of expressing what it is I wanted to achieve, the process became a lot more fluid and I was able to get the results that I wanted with something like this. I find that creating the character sheet and also putting as much detail in the character sheet as possible. Initially I just had a ghost skating and he had some blue shoes on, but I didn't have these small details like the board. He looks like when he's up in the air, board on his arm, close up of the shoes, close up of that of his eyes, particularly that was something that kind of that didn't work across a lot of the images and videos.
But I find that creating the characters as I'm creating the scenes, which is something that you can't do in a traditional production. Usually you would have to cast all the characters and people. And with AI, it does allow you to work on a section. And I just prefer to do it like this because working with AI, it's an iterative process. You produce one thing and then move on to the next step.
But, and only once you're happy with this, move on to the next thing. And I just find that if I had all the characters and everything, and again, this is just my process. I'm not saying it works for everybody, but I did find that keeping it in isolation and moving from character to character as I was going through the scenes, as opposed to doing everything upfront really did help. For each of the plates, let's call them plates. They are just the images used to guide the Seedance renders that I created. That was the video model I used, which is Seedance 2.5. The images themselves, in order to get consistency, I used pretty much the same structure.
Something that really does help when you are creating. So I was using GPT Image 2.5. I do find the GPT sometimes does create kind of fractal weirdness. It's difficult to explain exactly what it is, but I found that using this just at the bottom here as a global sort of forbidden list of negative prompt of what you don't want to see in the images.
So any weirdness that goes on, they really did help get a consistent look across the images. And remember that the images, when you are producing something like this, are the guides for what the video can look like as well. So using this structure across every single image, an example here, using the clay render, which came out of Blender, this as a consistent style across all the images, which I created in the first version of the video, and then what it is I wanted to create along with the description of what's getting created really just helped me remain very consistent there. When I moved over to video, it became a similar process, but Seedance accepts slightly different shape of what the prompt needs to look like. Traditionally, when I say traditionally, when you started out making AI video before Seedance and Tling came around, you used to start with an image and you would animate that image.
So you would have a starting image, but everything about that image, that would be the starting point and then the video would continue from that. Seedance allows you to add in references. Which to me is just way more creative. And you also get so surprised in a great way by the results that come out.
So using the same structure and using reference images for the characters. So you can see here, I've got white ghost, white details, the pink ghost, and then what the environment is without placing a starting frame in there. I was able to get such rich results because Seedance just has this capability of doing that. Also breaking each shot down into a little bit, it wasn't necessary for all the shots, but like from an example here is like from zero to two seconds. And this is placed inside the prompt from zero to two seconds.
This must happen. Then let's say you're creating a 12 second clip, two seconds. Then the next two seconds, this must happen. The next four seconds, this must happen. And explaining that, look, I'm not writing these prompts out by hand. I'm using Claude to explain what it is I want to happen. And Claude, based on the structure that I gave it said, okay, cool.
This is what it needs to do. We'd plug it directly into Seedance. He would actually write the prompt out for me. I would then take that, place it into FLORA and then run the iteration, wait for it. Once I was happy with that, move on to the next one.
But Seedance is an absolute rock star when it comes to the sort of creation. One of the shots in that film, I wanted to try and create or try and get as much of a continuous camera motion shot just to see if it was possible. I'd seen quite a few videos around of where people were using Blender animating in Blender, just so I don't know Blender at all. I've used it. I've dabbled with it a couple of times, but I don't know. I would never know how to do this without Claude.
So connecting Claude via MCP to that and then saying, telling it this is what I want to create. It then created a, which we'll take a look at shortly. I created the full sort of animation of how the ghost rode towards the railway track. And I wanted the camera to kind of swing around, land in front of him.
And then he turns around and moves away. I could have done that directly via Claude directly into Seedance, but I really wanted to see how much control I could maintain and how much consistency I could maintain by using Blender. And it worked out really well. Some of the things that I found challenging, as I mentioned, I started the first version of the video. I was using chat GPT, copying the prompts. I have a custom GPT, which I've written to help me write Seedance and GPT Image prompts. And I was copying, pasting into FLORA and then kind of running everything from there.
And I just found that once I moved to the Claude sort of option and everything plugged in there, Claude is just, it's great for that sort of thing. And the fact that you can plug FLORA so beautifully into it and it connects so nicely with the canvas and just, it's just, it just makes for such a, yeah, just a great experience in terms of directing. So I'm directing Claude, telling it what to do and it's just doing the things. And then at each step I'm telling it, okay, this is what you need to do.
So moving from chat GPT to Claude was definitely something that was, yeah, very beneficial. The resolution of the videos in terms of Seedance can produce up to, I think it can do 4K or 10 ADP. So that is just the quality of the video. I thought I'd found a hack by, I'm a bit of a like squirrel when it comes to like saving credits. And I think a lot of people that do use these tools that resonates with them because you've got to set an amount of credits. I don't want to use up all of them, right? And a 720p video costs X amounts depending on how long the video is.
I thought what I could do is I could create all these videos in 480p and then upscale them. But I wasn't quite happy with the, and these I think is just my ability. Nothing wrong with the tool. It just going from 480 to 720, I could still see a bit of inconsistencies that I wasn't happy with.
So I ended up just going straight to 720. When creating a, for example, like a 15 second clip. Creating that clip when there are cuts inside it is great and to save yourself some time this is something I worked out which seems obvious now but if there is a scene between seconds four and seven and you've got that three second or four second clip that you want to you only want to isolate that and you only really want to re-render that so that it yeah so you don't have to go and recreate the full 15 second clip which costs money and takes time to render you just take a snapshot of that tell Claude look this is what I want to recreate just that scene use the same references use the same text prompt and just generate that scene for me as opposed to running the whole model by itself but we're running the whole prompt again I did test a couple of model variations I used WAN 3 Minimax 3 which are amazing models they just they pale in comparison to what Seedances I have to do and I wasn't able to get consistency so I just stuck with that this is just a note on story if you are ever doing something like this I found that I every video that I create I feel like I need to use it I'm just like I've rendered this it's caught cost me tokens so like I must put it in but sometimes it doesn't fit the story so just be honest with yourself also run it by friends and family and if they'd be like maybe you should take that out that's not really working but the point I'm going to is don't be worried about taking things out of a story that aren't really working I thought this is just quite interesting in terms of how long this took the number of prompts that were run how many video generations this is the sort of cost and spend on flora how long it took and then the tools that I use to put it all together this 35 hours I think is a little bit shy of what actually happened I was doing a lot of editing in After Effects so didn't really taking take that into account but anyway I thought that's that was quite useful to check out there okay so before we go and dive into what it is that I produced I thought we take a quick look at what you can do in flora so this is an empty canvas you can connect it to yeah your favorite large language model and have a conversation there but if you are working manually directly into it you have this is a node based application so you have a little kind of addition button there you can create text image video audio whole bunch of these so if you click on text and this is what I really I truly love about an application like flora is you've got all these models built into so you don't have to go to chat GPT you don't have to go to Claude you have to go to cling or the Seedance website or whatever the case is everything is built in here entirely so I say a ghost skating in a park that is the prompt I then take the run it out hit image I'm just gonna use the new nano banana model which has just come out I can set the aspect ratio I can set the resolution can do whatever I need to it also tells me the cost there hit that we let that run so while that's rendering once this image does get produced you can then do a multitude of things to it or with it this is a just a very basic explanation so once this image is produced let's say I wanted to create a video from that link it out to the video here is just an abundant list of tools that you can use and test look my personal preference is obviously the as I mentioned I used to Seedance for this but you've got the ability to pretty much link to any of the tools that are available in there so I cannot edit this image manipulated do whatever I wanted to with that okay so that is just a very basic idea of what a canvas would look like just a snapshot of what my final film look like I mean this might look like a spaghetti junction of how for anyone who hasn't seen it before but we're gonna talk through it but that is yeah that was the sort of final film and what it all looked like before we get into that I just want to show you a where I started okay so I dropped this image in here initially I dropped it into chat GPT and I said okay create a prompt that's gonna allow me to put it into flora and then create the character sheet so this was the initial character sheet that I had and not enough I thought it was enough detail but it wasn't there were just too many inconsistencies in it and it's also only one image just again bearing in mind that these models can take in a lot of information so when you are connecting it to something like Seedance having more information is actually better provided that you have curated exactly what it is you want to produce so that came as the that was the sheet and then just to show you the sort of inconsistencies I was getting there's the ghost in this kidney bowl thing but now if you look at this image here, you can see the shape of the skateboard park is like very different. This was a beautiful shot, but just again, kind of didn't fit in anywhere in the park itself. Again, this one didn't work, but I just, I went ahead with it just to see and test and get a feel for, what I was, what I was in for here, produce some prompts. Some of these shots that came out in, Seedance with the available images that I had looked really cool. I mean, it was super cinematic, but again, consistency of the bowl.
There was some cool shots, just read the closeup shots really worked, but for the most part, I think I ended up using this shot or it's a very similar shot. I think around the same prompt, in the end with the second video. So I shot scene one, scene two, also, getting consistency of the house. I really, it, it was just a case of creating a day and night shots. Yeah. I think I used, Seedance 2.
So using start and end frame here, I'll get a little bit more into that and why I did that there. But for the most part using Seedance 2.5, but there were little things like when he runs up to, I mean, if you pay like I could see his legs from underneath and I was just man, this is not working. And I was really, I started like bugging me because I'm how do I get the consistency? How do I really nail this?
So I took this canvas, I plugged it into Claude and I said, Claude, my friend, give me a hand. I'm struggling here. I said, let's connect it to blender. Let's build the skate park. Right. Which is exactly what I did over here. Let me just give you an idea of what that looks like.
So this is, as I said, I do not know blender at all, but this is what, Claude produced for me. Okay. So this is just the skate park itself. So I was able to use this and you can see all these dots and things that are around here are camera placement.
So it's placed the camera at different points. So I said, I want to top down shot. I want to close up shots. Just, and I said, surprise me with another 10 shots and it just went ahead and did all of that for me. What it ended because I kind of just extended onto all of this is it built the whole little town for me.
So there you can see the railway tracks. There's the houses. There's the cul-de-sac scene, which I didn't end up using anyway. But if you're ever going to do something like this, just rely on Claude or a GPT does a pretty good job as well to kind of run all of this for you. I don't understand 3d modeling, at all.
So it is really just, it's super fun and you can create worlds that are, yeah, just pretty insane. Okay. So back to the initial thing. So I said, Claude, make the park, which it did, and then create a whole bunch of shots.
So you can see now it's rendering these from Claude is rendering these directly from blender and it automatically pasted this into FLORA. Okay. So I pasted all of these and then started rendering based on the, visual references that I gave it. I said, look, this is what it looks like at night.
This is what it looks like during the day. And just went ahead and started building out, the scapegoat. And this is when I really started feeling it. I was just okay, this is working and this is looking really cool as well.
So I went ahead there, but then eventually I stopped. I said, look, this is not working. There's too much going on. Let's create so that here was the, just the additional character sheets. Everything was kind of all over the place. I created some agency guide, which wasn't really working.
So I said, okay, this is not working. Let's start again. So we created a new project. Claude actually just set the whole thing up for me, which is pretty amazing. Same start image. He has my storyboard.
And then what it also did is just left some notes in case I wanted to refer back to them. This was just in terms of what the prompt structure to be for Seedance and for GPT Image. And now I was able to, and this is also another thing that I really love about FLORA is the ability to just kind of section off a whole bunch of nodes. And then you've got access to all of that. I mean, there is a lot of, these are being used in a lot of references, but you've got all access to this.
So this is the character lock. Cool. You can actually lock it off. So, and if you're working in teams as well, this becomes super handy. If you've got characters that one person's doing the characters, one person's doing the environment, and you can work separately on the same canvas.
So here's the characters. This was the initial one. I said, okay, create turnaround. Sheets, which the ghosts eventually formed as well. He created the initial bully. He, yeah, I wanted him to be a little bit more intimidating, gave him black hair. Some of the other characters who didn't really feature in there.
And then this was the sort of cul-de-sac, where he skated through the cul-de-sac, which I actually cut from the form. So that is all of those. Then I had this sort of blender rendered box over here, which was just the shots, the snapshots of the park. Of the roads, the top down shots of what it looked like. There was a shot as he skates at night out of the park, over the railway tracks.
So these are all some of the renders here. But how I went about this is, you would think, okay, cool, well, you just told, like all you did was told Claude to do the thing and it did it for you. It wasn't like that at all. I made sure, and I mentioned this earlier, trying to be iterative over, taking an iterative approach in terms of how you are creating the scenes.
So my process, was saying to Claude, this is, give me the shot. So show me what the shot might look like through the fence. I want a night and day shot. So to produce that shot, once I was happy with that, I then moved on to the next shot, as opposed to saying give 10 angles of this and from the, of the skate park, of the house, of everything. The problem with doing it like that is if you're not looking at each image individually, it's difficult to spot inconsistencies for one and little artifacts that might pop in. Like one of the shots that kept putting black foliage and trees inside the park, which obviously didn't work.
So I stopped it then said, okay, reproduce that please. And kind of, yeah, just do another pass on all of those. So here you can see that being the blender plates, that being the night plates, day plates. Perfect consistency every time the lighting's the same. Just, yeah. Anyone who's worked with these tools before will know that this just does save so much time and so much, rendering and guesswork because you don't have to guess anymore.
So there's the ball different times of the day. There's the night shot. What you will notice about all of these shots though, is none of them have characters in them. This is an important thing to know because if you, I just found that if, let me show you an example here. If you look at a shot like this, okay, I use this as a reference image in C dots. It's the character itself inside the scene.
Now you would think it makes sense to put the character in the scene and then animate the scene, but it doesn't work. You can try to get it to work like that, but you're fighting against what Seedance is really good at, which is taking rinse images. And then you're expressing via the prompt, what it is you want those images to do, what you want the character to do, what the environment needs to look like and how the characters interact and engage with that world. So I use these as a references for what the environments look like.
So building up the references and what's so great about something like this is you can so easily just go back. I've also placed these into kind of how chronologically I was going through the scenes, which again, just, that's just my process and it did really work. So got all these scenes here, multiple variations of that. And then I started with, I started with scene one and then started building each thing as I went along.
So here we have scene one, scene two, three, and four. So let's look at scene one, how I organized this to a scene one. That's shot one, shot two, shot three, and shot four. So I did shot one until I was happy. There's some images in there.
And then you kind of just take the shots and look at the ones that are working. Like I took some of these shots that Seedance produced. I did this quite regularly. Here's the opening shot, but the opening shot did not necessarily have to be part of the second shot.
So I often find myself, this is a 12 second clip. I produced this multiple times. Think of it as doing multiple takes. If you're, behind a camera, you shooting the scene, it's cool guys, run the scene again, run the scene again. You're picking the best shot from that's exactly what you're doing here.
So, and then kind of editing, I'm, I just, this is just how I work. I'm kind of editing as I go along. So in my head, I'm okay, that works. That doesn't work. Look at the clip side by side and then slowly decide, okay, cool. Just something I want to add in.
And then actually pull it out of downloaded from FLORA, put it into the timeline in After Effects, whichever tool you decide to use, and then do a little bit of editing. It's very easy to get a rough cut from this process because the videos are just like so freely available there as well. So. Couple of scenes from there and then, yeah, just went ahead and built all of these through there as a, an afterthought. I wanted to introduce the ghost character, just so it wasn't this random character that popped up in the end.
So ended up using, one of these scenes here, which was just a closeup shot of kind of behind the ghost, all the pink goes kind of watching her, yeah, watching the white ghost goes through the skate park. This scene was pretty straightforward. For some reason, it was just a, it was like a locked off shot. I found that the continuous shots where it's this is a 12, this is a 10 second shot.
This is what needs to happen. And this is me explaining to Claude, right? Saying I want this to happen. Use this prompt structure, which it's already knows because you've told it at the beginning, always use this prompt structure. At nighttime, this is what the lighting must be like. During the day, this is what the lighting must be like.
And then it was just a case of just banging these out, which yeah, which worked pretty well. Where I did, and this is something to remember, where I did end up using Seedance 2 and not 2.5. So just to know, well, just so you do know, Seedance 2 is slightly cheaper than 2.5. And for some scenes, you don't really need to use the most expensive and the most powerful model. A scene like this, for example, this was just and I used start and end frame for this.
So it's just understanding that sometimes you can revert back to a model that is, you don't need any craziness going on here. And this was literally time lapse through the night as the sun rises at the end, using the start frame as the nighttime of so creating the two plates based on what was produced in Blender and then rendering them out as day and night. And yeah, this just worked out perfectly. So, and then the second shot in that scene was him coming out of the door.
So it was kind of moving from that to the shot over here. And yeah, that worked out really well. The next scene was again, like rocking up to the park, coming back. The scene that I did delete was the scene. It just didn't feel like it was really part of I tried to introduce this family who saw him and the little kid kind of sees him because he's a ghost. And yeah, I mean, little kids like ghosts, right?
But it just didn't fit the scene. There were some cool shots in there. But again, I rendered it a couple of times, but just sadly just had to let go going back to the park. Again, multiple renders, I picked pieces out of certain parts that worked and sort of certain parts that didn't.
Then it was a case of just picking the best shots. This one definitely made it into the final cut. See, this one didn't because the angle was incorrect. Again, I produced everything.
And then I was he's looking at it from the wrong side. These are just things that you need to kind of pay attention to or show it to someone else so that they can see that it is incorrect or not. Getting the ghost to leave. And then I needed a transition scene of when he left the park and then when he came back. And I wanted to use that as an opportunity to link to Blender, get Claude to link to Blender, create a camera movement, which was what I expressed to it. And this is what the camera render looked like.
So this became the reference that I could use inside a Seedance node. So if we play two of these together, let's try to do this here. So you can see it's kept the exact camera motion. Ghost does exactly what I wanted to. It's just such a peaceful process. When you get, when you're able to achieve something like this, I really, I was just so stoked at how this came out. And it's available to everybody.
It's amazing. So that was based on that. I then also wanted to show you. This one, because this took, this was a lot of people interaction.
So we had the ghost interacting with the bully. We also want to, I also wanted to introduce the, the pink ghost kind of skating through the bully getting up. There was a couple of inconsistencies then in terms of the board that he was using, because I did find, you'll notice here, the underside of his board is green, but the underside of the pink ghosts board is actually green. So it was little things like that where I'm Oh, the shots perfect, but now I have to re-render it. Because yeah.
And then it's again, saying to Claude, please recreate this prompt to make sure that you're using the correct reference of the skateboard itself. A couple of other inconsistencies was when we had a little ghost buddy here, kind of squaring up to, yeah, it was just, there was lighting inconsistencies. This, I mean, this just kind of looks like it was stuck in. The baddie wasn't, yeah, I mean, this was, if you see how he's towering over him there, I mean, there was, and then, yeah, just didn't work.
So I fought with the scene. It took me longer than it should have, but at the end ended up just using one of the scenes that I got kind of out. That was a 12 second shot. What you can do too is, let me just go to the shot.
So here is you can take, Seedance and use it as a reference. I plugged this, I took this video, which was pretty much perfect. I just didn't like the way that the ghost fell. It looks like he just stepped back.
So I just thought as a creative choice there to, as he's falling, do a top down shot. So it looks like he's falling better. So instead of having to recreate the entire shot, Seedance has the ability to, you connect it to another Seedance node. You say, use the video that's connected as a reference and show me this exact video. Don't change anything from multiple angles.
So what it was able to do then is it did this top down shot here. So that was the shot I needed because it kind of broke the fall for the awkward fall that happened there. So that worked out really nicely. I got that in one, which was great.
And then just going through something else that you can do, it took me a while to work out exactly how to do it, is you can extend a shot and FLORA allows you to do this wonderfully. This shot I didn't use, but as he skates in from here, I use this at the end. So I can play to this point, stop at that point, and then extract a specific frame. This is very handy if you're trying to get consistency and continuity or a continuous shot from one single point.
Okay. So this, if I say this frame, I hit that, this frame here. It'll take a while and then generates the frame. So you'll see these two frames are exactly the same.
So what I can do now is I could use Seedance to perhaps or connect to Seedance again and say from this point exactly, this is what I want to happen. So this allows you to go from let's say you've only got a 10 second shot, but you want to extend it to 15 or 20 seconds. That's one way of going about it. Another way of going about it, which almost like seamlessly blends the frames together is I took this connected to another reference. Sorry. I connected to another Seedance instance. And if you've got a video connected to a Seedance video as a reference at the, you've got the ability to change it to editing or extension.
So editing, what you can do is say, please change the color of the, this goes to blue and change X to whatever. But extension, what that allows you to do is extend the video. So I, again, Claude helped me with this prompt here, but it was extend this video by 10 seconds, continuing seamlessly from its final frame. And that allowed me to skate into the bowl, get to the point where it kind of camera goes straight up to them. I needed a transition shot from like daylight and then it switches tonight.
This didn't really work because they were skating in the same direction. So it looked kind of okay, but it just felt like it stopped and then moved away, which I didn't really want. So ended up creating, and this is what I was referring to right at the beginning about, you can just clip out a 10 second shot. So this was just, this was an eight second shot, a continuation of, because I needed that black frame with, yeah, the dark frame to switch from day to night.
Let's get it off. I actually ended up changing the, yeah, change it to a horizontal version. Final shot there, looking directly down, slow zoom out. And that my friends is all she wrote. I hope you learned something today.
