What ligand you design to depends on what you think is important for catalysis. Probably the main feature is transition state stabilization (hence the tutorial). Depending on your reaction, you can probably get decent results with just modeling the transition state – I believe that the published Kemp eliminase designs used that technique. Sometimes more is needed. For example, while early Diels Alderase designs had great transition state binding, the pocket appeared to be too closed to permit substrate binding. Designing to a ligand with bond lengths intermediate between the (non-covalent) substrate distances and the transition state opened up the pocket and gave better results. An extreme example of a “hybrid” structure was used for some of the retroaldolase designs. For those, a number of structures (substrate, product, and two transition state mimics from different steps in the reaction) were simply superimposed into a single structure, so that each atom was present in three to four different locations. It wasn’t a molecule that could ever exist in nature, but being able to use such theoretical hybrids is a benefit of computational design. — My suggestion is to look at your reaction, and include those features you believe are important. If you think TS stabilization alone is sufficient, then just go with the TS mimic. If you want to include substrate/product binding, add atom locations for the interactions you want to stabilize/favor. Keep in mind you can do several runs with slight different parameters each time.
To use the enzyme_design application (see documentation http://www.rosettacommons.org/manuals/archive/rosetta3.2.1_user_guide/app_enzyme_design.html) you need to have a starting structure which not only has the scaffold protein, but also includes the ligand/TS mimic/hybrid you’re designing to. This needs to be in approximately the same location that you want it to end up in. (Particularly, you want to place it such that your constraints are all satisfied.) As you have a model for where the phosphate of the ligand should end up, my recommendation is to open up both the template model and the ligand pdb (specifically the ligand PDB that is the output of molfile_to_params.py, as that program renames the atoms) in something like PyMol. You can then use the pair fitting wizard (or equivalent) to move the ligand pdb structure such that the phosphate atoms line up with the locations where you want to put them. Save the translated/rotated molecule, and then use a text editor to create a PDB file with the template protein and the translated ligand. Also, as you have constraints, you’ll need to specify how the constraints in the .cst file match up to residues in the PDB (see the info about the “REMARK” line in the enzyme_design documentation). This is the PDB file you want to use for input to the enzyme_design application.
As for which constraints to use, use the constraints that you want to be included in the output structures. From the perspective of the enzyme_design application, constraints are just a way of saying “I want this general geometry to hold in the output structure”. The enzyme_design application will then try to keep the constraints valid throughout the design. So you can’t make a constraint to a non-existent residue, but you can make a constraint to a residue that’s slightly out of range, although it’s recommended to have the constraints be (almost) satisfied going into the enzyme_design application.