After successfully spinning the flywheel and pressure plate at 9000 RPM, I felt confident that things were looking good with the motor. I re-installed the clutch friction disk and bolted on the transmission. After spinning up the system, it started rattling and skittering across the floor around 5500 RPM. Geez. This is a royal pain.
After pondering this a bit, I tried the following experiment: I unbolted the transmission, removed the friction disk inside the clutch that attaches to the transmission shaft and bolted the transmission back on. After revving the motor up to 5500 RPM, whole thing still vibrated.
So, by just spinning the motor, flywheel and pressure plate at 5500 RPM, I get no vibration. By simply attaching the mass of the transmission (again, no friction plate in the clutch), I get vibration. Resonant frequencies anyone? I'm guessing that the added mass of the transmission brought down the resonant frequency of the system from above 9000 RPM down to 5500 RPM. There also may be some loose parts in the transmission that resonate at 5500 RPM and cause the assembly to vibrate.
I'm somewhat uncertain what to do at this point. I can put the car back together, program the AC motor controller to limit my RPMs so I never get into the resonant zone, and drive on with reduced performance. The other option is to find a new transmission (or borrow one) and see if it has the same resonant problems. Given all the effort and money I put into this transmission, I'm not looking forward to dealing with another one.
I suppose I could try adding some resonant dissipation material (like a dynamat) to as many places as I can on the motor/tranny to dampen things out. I don't know if materials like this would be able to dampen things enough. I'm probably going to sit on this for a bit... Rats.