When starting to record it is important that you use a click track if you are planning on editing or using midi with your tracks. If you are recording a jazz quartet or something a bit looser it's best to avoid the click track in order to save the musicians sanity.
Signal Chain
First you need to think about the path from your instrument into your audio device. You can have many things going on but the more you do at this stage limits the amount of control you will have once in your audio program. I recommend just using a bit of compression and if you have a nice preamp running through it at this stage. Only use effects if you can't get the similar sound with your software.
Here is a basic chain of events once your instrument is recorded into your audio sequencer:
EQ (EQ is a complex and powerful tool and is often overlooked. You can use EQ to tailor the sound of your instrument and decide in which frequency spectrum it will preside. If all of your instruments are fighting for the same frequency you mix will be muddy and flat.
Compression (be careful here because you can easily overdo this step, if you already compressed your signal when you recorded your instrument then just use a subtle amount here or even bypass this step.)
Effects (this is a great spot to put anything from weird effects to amp modelers to reverb. If you are using both effects/modelers and reverb place the effect first and the reverb second.)
Limiter (optional and usually put on the Main Out of the audio sequencer to get the fullest sound without peaking.
Advanced Techniques
Side-Cain Compression
This technique is great as either a way to make a certain element stand out or as an effect.
Side-chain comp is commonly used for presentations where Dialogue is important and music is in the background.
I've used side-chain comp for bass solos since bass is a muddy instrument. A little bit of side-chain comp seems to help bass solos pop out of the mix.
Since I mostly use Ableton Live This example is about using side-chain compression in Ableton.
Summing Tracks
Summing tracks is a way of bringing a mix together and can be a great organizational tool. The following is how to Sum a mix in Ableton:
Create four audio Bus tracks and label them Guitar Bass Drums and Keys (this is just an example and can be whatever instrumentation you desire)
Send each track to the logical bus e.g. guitar goes to the guitar bus etc... Make sure you select bus only on each track so that it only plays on the bus.
Now you can easily change the levels of each group with the buses and tweak each instrument accordingly
You can add effects like Compression, Limiters and Reverb to each bus for a fuller mix.
USING LOOPS AND SAMPLES
Loops and Samples
Samples and Loops although never as good as the real thing can be very interesting and fun to track with. I often use loops just as a reference for a song, in a way it is like storyboarding an animation, you lay out the rough concepts and then bring in the real detail later.
Ableton is king for using loops and samples also Reason and Sony Acid are very good.
Tricks in Reason
Creating an Aphex Twin style Effect.
Freezing Audio
Freezing is very important when your computer starts to overload its CPU. To freeze a track in Ableton all you have to do is right click the track in the track view and select freeze track. In Nuendo I think you select a tab in the left box which looks like a star, you can always see the programs help doc and search for Freezing. Freezing bounces the selected tracks and uses the bounced audio instead of original, this disables any VST instruments that were on the track so you can no longer edit them unless you unfreeze the track. To unfreeze it should be similar to the process of freezing e.g. in Ableton right click the frozen track and select unfreeze.
Mixing
Mixing is an art in itself and I don't claim to be great at it however I can provide some basic concepts that may help you out:
Think about dimensions of your sound, where should everything be in the mix, distant or close.
Think about the frequencies of each instrument and try to figure out where to cut and boost each one to fit best in the mix.
Sum tracks as mentioned above
Side-Chain if needed
Get the mix sounding nice on your monitors and then listen to it on a home stereo or even laptop speakers, take notes of things that come to your attention, is anything buried or jumping out?
Mastering
My best suggestion is to get this done professionally as it requires lots of knowledge and decent equipment for best results. However if you don't have any cash to do so, try messing around with EQ, Compression and a Limiter on the Final Mix. Try to be subtle, and don't squich the mix too much.
