Jonathan found that Maya was able to perform camera projection and export the resulting texture. The manual process for setting it up was as follows:
Once the projection was configured, you could move on to the actual projection step. You still needed to shift-select the target mesh and shader, then find the Convert to File Texture button in the Hypershade window, where you need to configure the settings for the export.
As you can see, it's a rather tedious process.
The issue we faced was that a single export was very slow due to the large resolution we were rendering it in, not to mention the setup speed. This process would need to be repeated many, many times for painting iterations, so I was very incentivised to automate it.
The main goal for the script was to directly convert the process into code, step-by-step. Most commands of the process could be found quite easily within the maya.cmds documentation, but not the export command itself.
I started by replicating the node setup process in code. This first bit is what happens when you click on the "Create as projection" button mentioned earlier.
def projection_create_nodes():
# Create necessary shading nodes
fileNode = cmds.shadingNode('file', asTexture=True)
tex2dNode = cmds.shadingNode('place2dTexture', asUtility=True)
tex3dNode = cmds.shadingNode('place3dTexture', asUtility=True)
projNode = cmds.shadingNode('projection', asTexture=True)
# Make common connections between place2dTexture and file texture
connections = ['rotateUV','offset','noiseUV','vertexCameraOne','vertexUvThree','vertexUvTwo','vertexUvOne','repeatUV','wrapV','wrapU','stagger','mirrorU','mirrorV','rotateFrame','translateFrame','coverage']
cmds.connectAttr(f"{tex2dNode}.outUV", f"{fileNode}.uvCoord")
cmds.connectAttr(f"{tex2dNode}.outUvFilterSize", f"{fileNode}.uvFilterSize")
for i in connections:
cmds.connectAttr(f"{tex2dNode}.{i}", f"{fileNode}.{i}")
# Connect the file texture output to the projection node
cmds.connectAttr(f"{fileNode}.outColor", f"{projNode}.image")
# Connect the place3dTexture node to the projection node
cmds.connectAttr(f"{tex3dNode}.worldInverseMatrix", f"{projNode}.placementMatrix")
# Connect the projection node to the material node
cmds.connectAttr(f"{projNode}.outColor", f"{LC_Projection.targetMaterial}.color")
Afterwards, it boiled down to more connections and some editing of node settings.
# Set diffuse value for material to 1
cmds.setAttr(f"{LC_Projection.targetMaterial}.diffuse", 1)
# Set wrap UV to false on the texture2D node
cmds.setAttr(f"{tex2dNode}.wrapU", True)
cmds.setAttr(f"{tex2dNode}.wrapV", True)
# Set painting file path on the file node
cmds.setAttr(f"{fileNode}.fileTextureName", LC_Projection.paintingPath, type='string')
cmds.setAttr(f"{fileNode}.filterType", 0)
# Set projection type - perspective, fitFill - fill, fitType - match resolution, and default color to black on the projection node
cmds.setAttr(f"{projNode}.projType", 8)
cmds.setAttr(f"{projNode}.fitFill", 0)
cmds.setAttr(f"{projNode}.fitType", 2)
cmds.setAttr(f"{projNode}.defaultColor", 0,0,0)
# Connect the camera node to the projection node
cmds.connectAttr(f"{LC_Projection.targetCamera}.message", f"{projNode}.linkedCamera")
I found this
blog by Andy Davis to be very insightful during this time, and I suggest taking a look!
While I wasn't able to find a maya.cmds command for the export step, I was able to find
one in MEL script by echoing the command, so I called it as is in the script using mel.eval().
I also added a few layers of validation just in case I missed something, or if another team member wanted to bake something too.
Here's a quick example of a function validating the selected camera. To keep track of validation results, I stored them in my dataclass "LC_Projection", which I used to organise global variables like validation booleans, UI colours and filepaths.
def validate_camera(*args):
try:
cmds.camera(LC_Projection.targetCamera, q=True, focalLength=True)
except Exception as e:
print(e)
LC_Projection.camera_validation = False
cmds.iconTextButton(LC_Projection.cameraStatus, e=True, i1="error.png", ann="Camera is invalid.")
change_project_button()
print(f"{LC_Projection.targetCamera} is an invalid camera.\n")
return
LC_Projection.camera_validation = True
cmds.iconTextButton(LC_Projection.cameraStatus, e=True, i1="confirm.png", ann="Camera is valid.")
change_project_button()
print(f"{LC_Projection.targetCamera} is a valid camera.\n")
Another major problem that we ran into was that moving the camera or objects in the frame would break the illusion, since the texture projection only affected camera space.
The original plan was to use Blender painting method to repaint the problem areas as they appeared in shots.
While this seemed like a perfect solution on paper, we ran into the
roadblock mentioned in the anamorphic section, where we couldn't match the Blender camera 1:1 with the Maya/Unreal camera.
The second idea was to run a texture projection on the start and end of the moving shot, then blend the two images. This seemed like a sound idea at first, but I realised that I had no clue how the blending would actually be done.
The manual way would be to add the two paintings as two layers in Substance Painter and paint a difference mask by hand, but it seemed like too much of a laborious process at the time.
In hindsight, this may have been the most efficient method, had we been able to use this material for all shots in each scene, but we would lose a substantial amount of creative freedom in the process.
At the same time, it would have added several more shots to the environment artist's workload, which we could not spare.
Later on, I thought of using a mix node in Unreal to blend the two angles using camera projection as a driving mask, but I couldn't get it to work the way I wanted.
For hero objects that moved a substantial amount and weren't too large, we would get the character texture artist to handpaint the textures in Substance Painter. Objects like the battery receptable and the elevator would fit in this category.
For any other objects, we couldn't come up with a more efficient solution than going in ourselves to repaint the exposed sections in Substance Painter.
With all basic aspects of the general texturing workflow explained at this point, I can now outline the process. The process was back and forth between me and the environment artist, and looked like this:
I took High Resolution Screenshots of lighting, and any black/white masks required for the shot to help isolate smaller objects for painting, then sent these to the environment artist to paint over.
Once the paintover was done, I ran the projection script in Maya to generate the shot texture.
In Unreal, I imported this projected texture and assigned a keyframe on the matching shot to switch the environment's material to this new texture. I made many material instances of the base shader to do this.
One hero shot per environment was prioritised, so that the resulting projection could be used for a base for the rest of the shots in the environment.
In this example, the main establishing shot for the lab environment was used as the first base.
As touched upon earlier, I sent screenshots of the first lighting pass, along with a few black & white masks to use as clipping masks.
Since the projection only affected what the projecting camera could see, I needed to perform separate projections that each focus on major objects which were obscured or obscured other objects.
Thus, the painting was sent back to me with the masked objects present on separate images.
Then, I used the projection script to perform separate projections for each of the individual objects that are obscured by other objects.
After I imported the textures into Unreal Engine and applied them to the mesh, the colours could be loosely carried forward to help paint subsequent shots in the environment.
One hero shot per environment was prioritised so that the resulting projection could be used for a base for the rest of the shots in the environment.
It goes without saying that this process involved a lot of back and forth feedback and refinement, including shot changes midway, so the projection script ended up seeing a lot of use.
Please visit Wen En's artstation posts
here and
here for the shot paintings!